HiSeq2000 - Next Level Hacking


#62

Hi @kaspar
Yes, this is very scientific :slight_smile: The two cameras are connected to the electronics by only quite fragile flat wires, so best to keep the tape that I put. Once you have the frame grabber card, you need to connect the camera to the card with the cable that @tboysen sent you. Then find a way to power the camera with +/- 15VDC and 5VDC (about 0.5-1 Amper each). With the Hamamatsu HCImage software ( https://dcam-api.com/hamamatsu-software/ ) you should see both cameras and be able to take images from both cameras. If you switch to TDI mode you need to select “internal trigger” (on the HiSeq the FPGA generates the trigger). Remember you will only see some lines as this is a line camera - you should see changes in brightness if you cover the slit :wink: . I tested this and it worked for me. And then you can start playing with drivers.

Have fun.
Urs

PS: ChaosLooper is also on it’s way.


#63

Welcome to the forum, @RSpanish ! See post by jmarkham earlier in this thread for a comparison of HiSeq2000 vs HiSeq 2500.
Urs


#64

Hi @RSpanish, regarding possible contributions, I can propose some things that I think would be useful, but I will leave it for others to comment. Also I’m not sure if you have students/time available so I’m just making this up as I go…

A diagnostics program would be handy. By that I mean something that can find out what drivers and hardware are installed on an instrument and then talk to all the different components to ask them what their firmware version is and see if they respond appropriately when asked to do some kind of action. The machine I have was retired after it failed and something like this would have been useful for troubleshooting.

I’m keen on controlling the microscope using micro-manager (open-source fully functional microscope control software). That would require device adaptors in C++ to control at least the microscope parts (x,y,z,lasers,cameras). In principle it could control all manner of things if it had the device adaptors. MM can also just be used as an API by other programs.

Regardless of whether MM is ultimately used, both the diagnostics and the program that drives the instrument could call low level functions that talk to the hardware - mostly via RS-232 with vendor-specific command sets.

John


#65

Hello All! I am a fourth year undergraduate student at a university in the USA. Chiefly, I just wanted to let you guys know that the information that you guys have been posting is SO INCREDIBLY helpful for my Instrumental Analysis course. I am so lucky to have stumble upon the Hacketeria website where I was able to find a block diagram of Illumina’s HiSeq instrument and images if the instrument components itself. This data has been so hard to find. Moreover, I have been having trouble finding information about the fluorescence that is used on the nucleotides. Does anyone know what is used?


#66

Hi,

The excitation is done at 532 and 660nm, from a discussion about dyes
some months ago i remember these dyes were used in a Illumina 4-channel
SBS kit:

Alexa 647-dATP
Atto532-dGTP
Dy681-dCTP
ProprietaryDye-dTTP

The last dye i don’t know, should be found out by seaching in the
patents Illumina is citing. Hope this is correct…

Teide


#67

Ok, @kaspar @jmarkham @bengtsjolen . For the HiSeq meeting at GaudiLabs, 8.-10. January 2019 (Tuesday-Wednesday) it is. I am really looking forward to us meeting around this exciting project and making an other step forward. I am happy to host you in the lab (have mattresses and sleeping bags) and in my place (very close to the lab). Plan your travel individually (you find GaudiLabs on Google Maps). We can later decide if we want to put the expenses in the crowdfunding budget.

Program suggestion:

- Make work the hacked machine
- Experiment with fluorescent scanning and other potential applications
- Think about software, interface and implementations
- Design crowdfunding and shoot a video

What else? What do we need to prepare?
Urs


#68

I have booked my flights! I will be flying to Basel in the evening on 6th of January I can take the train to Luzern or Zurich from there depending on plans. I’ll be off again on the 13th.


#69

@gaudi, I sent you an email to an address at hslu.ch. Is this the one to use?
Regarding the forthcoming HiSeq festival, here’s a couple of ideas:

In advance of the festival it may be useful to document some internals of HCIM and of Illumina’s program for driving the HiSeq using a couple of reverse engineering tools. Then, if we can run both programs under the control of a debugger while they talk to Urs’s HiSeq, we can hopefully extract information about calls to vendor dll’s not available from the serial port traffic. This is not may area, so if it’s yours, feel free to comment.

There’s an old version of the camera interface from MM here which, together with the examples in the SDK, could speed-up coding for the camera.

I have no idea what’s in the flow cell module on the stage - is there doco for that? From memory Teide had a patent but it was in German. Mind you, even in English I find those things hard to decode.

Does anyone have any suggestions for travelling from Spain (near Girona) to Lucerne? Specifically the relative merits of train, car and flying (which I’m not so keen on since it would still involve a bus back to Barcelona).


#70

Hello @gaudi ,
I am John(jmarkham)'s sidekick, I have question about the command lines extracted from Control software log file. Where does it come from? And if we want to control the FPGA board directly through Serial COM ports, where should we type it? Which software should I use, like Yet Another Terminal (YAT)? In YAT, we tried to write some commands to the corresponding terminal, either we got nothing or E3 from Phoenix Frame grabber. How could we test parts of hardware separately?
Kind regards,
James


#71

Hi,

@john, the german patent you remember is about the optical bench
mostly, it corresponds to the english one number US 8.481,903 B2, that
should be in the archive of documentation i provided you in the folder
“Paper and Patents” - or just enter this string in google.

The module on the stage contains components to securely hold the flow
cell in place and connected to the fluidic without leakage, via spindle
motors the whole thing and hereby the flow cell surface can be aligned
to the image plane, the metal plates holding the actual flow cells are
temperature controlled via peltier elements (water-cooled). I remember
some marking on the PCB about “magnetic compensation” that surprised me
a little bit. If some documentation is neeeded i can grub out the parts
and report.

For the beginning to reverse the machine’s control i think it’s not
necessary to do start with such timeconsuming efforts like digging in
assembly code. I remember the logs are very extensive when i hunted
down a bug in one HiSeq. I expect it’s enough to sniff the serial ports
traffic while performing specific functions of the instrument (in the
end all…). Important: To do it right it will also involve to think
about the messages that will only appear if the system enters specific
conditions like overtemperatures in components. An own control software
should be planned to act like a state machine without unforseen states
left. To find undocumented things the usual approaches like extracting
strings from the binaries are helpful, i don’t expect obfuscated code
here, but i’ve seen such things while reversing lab machines.

Sniffing can both be done on the control PC or - more reliable - with a
set of TTL serial adapters connected to the FTDI chips in the machine.
All information (beside the camera link) to and from the machine is then
catched - as i like it to express with Wilhelm Tell: “He must come
through this hollow way” :wink: I must admit that in the past i had not a
system with such an amount of (virtual) serial ports. If the timing is
tricky - i used a video camera to help syncing machine’s events and
message flow in the past.

And some remark - sending commands “just to check what’s happening” can
be risky! It is not invariably the case that the commands are foolproof
if you send them on internal connections or out of context, e.g.
activating a pump while the valve is not open. The same goes for the
components in the instrument, for example the Z-stage piezo actuator can
be destroyed if driven incorrectly - e.g. when a implemented control
loop hits a resonance frequency. And i heard of VICI valve controllers
bricked by sending the command to define the number of positions present
in the valve head.

Best,

Teide


#72

Hi James.
Welcome to the forum. YAT should work. After you start the computer, start the machine and wait for all the ports to initialize (as you would normally do to run the machine). Then start the YAT. Choose the serial port of the device you want to address (from the command list). For example the FPGA board is COM10. Then you need to set the right communication speed. For the FPGA board 115200 baud. Then send a command. For example “LEDMODE1 6” and the big light-bar on the machine should blink blue.
Note: The FPGA board sends responses on a separate port (COM11).

Urs


#73

I made a site with an email sign up (text adapted from @gaudi’s).

I propose the name “ReSeq” for this project. If there are no errors in the copy then we can put it up on a proper domain. Maybe reseq.hackteria.org? (just point the sub-domain at 104.198.14.52 and let me know). If you do find issues please report or correct through the GitHub project.


#74

Ok, cool. @dusjagr can you make the link?

Urs


#75

Printed a giant water flea to help talk to people about HiSeq hacking at 35C3

Printed the fluorescent cheek cells quite large too and made another poster with some explanation and a link (tbd):

(SVG)


#76

@tboysen, I take your point about only needing to look at the serial ports, but I’m a bit nervous about dcamapi and its calls. Having said that, the Phoenix framegrabber’s SDK is included with this test code here. Between that, the Hamamatsu SDK and the MicroManager Device Adaptor source I think we should have enough information.

I am the brick-maker referred to above and would second what Teide says about not sending speculative commands. Perhaps in the low level interface functions we can include sanity checks to intercept accidental bricking commands.

Teide, your suggestion of using an un-earthed laptop to talk to the USB port fixed all but two of the serial port drop-outs. Imagine my surprise and joy. We would like to revert back to the original machine though which raises the question of how to get rid of the offending earth loop. Low impedance connections between the cases?

@Kasper, we have been calling our machine the Franken-Seq (let’s just say that it’s far from what it’s creator intended) but I can see that for fundraising this may be less than optimal. Likewise Illuminati, NoSeq and Hide’n’Seq.


#77

hoi zäme,

i finally put that subdomain there…

looking forward to meet you all in jan!
marc


#78

Cool, we have a dedicated website for the project now.

Next week we will meet here at GaudiLabs in Switzerland for some HiSeq geeking and for starting off the development of a dedicated software. I am setting up the lab. Is there something we need to prepare in advance? @kaspar will already be here from Sunday on and we have time.

  • Maybe we could use some video or pictures of a sequencing facility for the crowdfunding video. @jmarkham @tboysen could you film something in your institution?

  • If you have interesting samples to scan, bring them with you

Urs


#79

Hi all,

i’m sorry that i cannot come, i’m unable to visit the meeting because i
have work in january that cannot be postponed. I hope the components
i’ve sent out arrived safe and timely.

I can produce video and picture material in my institute but have to
get approval before. What sort of footage you need? I can produce
broadcast-grade material in every format because i’m regularly helping a
film crew with technical things and so i have access to UHD/4K cameras
and postproduction equipment.

Happy New Year and success with your meeting! I’m eager to hear what
you are doing and accomplish.

Teide


#80

Hi all, I ended up getting a slightly earlier flight out of Barcelona and so could potentially get there on Monday arvo. Here’s my flights:

Barcelona (BCN) - Zurich (ZRH), Mon. 07.01.2019 LX 1953
09:45 BCN - 11:40 ZRH

Zurich (ZRH) - Barcelona (BCN), Thu. 10.01.2019 LX 1956
17:25 ZRH - 19:10 BCN

@Gaudi, does that work for you? If so I will investigate accommodation for the Monday night.

Regarding video of sequencing facilities, like @tboysen I’m happy to provide some footage, subject to approval from work. While I can’t video anything until I get back, once I know what I’m asking for I can start the approval-seeking process.

Regarding advance preparations, now that I’m on holidays in Spain I can skype at a reasonable hour of the day and am happy to discuss. It might be easier than typing.


#81

On the risk for bricking and how to mitigate it:

When reverse engineering and trying things unknown, it is always helpful to be able to restore the device to the condition it was before you started messing with things - ie dump all non-volatile memories so they can be restored later or having a working set of tools for doing firmware update etc with the same version that you have on the device and then also having confirmed that such an update would also reset or wipe configurations / settings or other data storage outside of the actual firmware. Under those conditions you can hammer it pretty hard without having to worry about breaking digital things at least - and when it breaks you just restore it and start again. This is actually much more helpful than just not breaking things since it gives you the confidence and freedom to explore much more freely and always be able to restore to a known state not having to worry about accumulated effects or memory of previous actions in the device. Unfortunately there is a chance of burning mechanical or other things of the physical world by sending broken commands still of course…

  • Serial memories, I2C or SPI, you can easily just hook up 3 wires to them permanently and be able to dump or write contents while holding the rest of the system in reset for example - beware of serial NAND-memories that require error correction and wear-levelling though - see below. EEPROMs and FRAM is a breeze and typically are used for configuration, settings etc even though less and less so (replaced by storing in NAND, eMMC etc to reduce BOM more often now)
  • eMMC is actually an SD-card so you can typically wire in a SD-card dummy adapter that you can put in a reader and make sure you only power the eMMC and then access it for reading and writing as if it was an SD-card. You can even remove an eMMC-chip and put a SD-card socket instead and then you can just change and replace the content by swapping he SD-card but even if this has been working it has been error-prone and resulting in weird bugs and such things that you just don’t want - and also it’s a bit tricky to solder on bga pads…
  • Parallell NAND-flash is a bit trickier for many reasons: needs 12-15 or so pads connected, and it is inherently error-prone so in order to be able to reconstruct broken data you need to understand error-correction used, and since it needs wear-leveling, locations where data is stored moves around so if you would move to another chip you cannot just rewrite the image because bad blocks wouldn’t match and as your chip ages it will deteriorate and things might have moved so you might not be able to rewrite the original image in a way that works. Typically easiest to dump/restore from within a system or a bootloader/u-boot or so which already handles those things for you as expected by the rest of the system - wear-leveling and error correction applies to serial NAND memories above also.
  • Parallell NOR chips you typically have to de-solder to dump and rewrite - separate address and databus makes it just too many pins and too complicated to wire it in(even though that has been done too, replacin with a socket e.g.) - so also preferably read/written from within the system. No wear-leveling or error correction needed on NOR typically so can be dumped and restored at will. Solder/desolder chips can be done a few times but typically the traces and the PCB give in after a few cycles and then you need a new PCB…
  • Most processors also have embedded non-volatile memory apart from the program memory - EEPROM or so - and typically this needs device-specific tools to access…

Once you have the ability to store and restore all non-volatile data you can break things at will and also create a library of different versions and configurations and be able to clone systems to try them out and explore them.