Digital X-ray: Engineering Necromancy

Since I am prone to mild graphomania, I will share my experience of engineering necromancy. You may ask, why necromancy? I will answer – it is difficult to call the resuscitation of a ten-year-old “hardware” project by any other word.

The project had many mistakes, both engineering and management ones, the “black swan” played a role, the budget, naturally, didn't come together, all in all it was fun. Perhaps this experience will be useful to someone.

Important: The text describes the situation in 2019–2020. Prices and the political situation have changed significantly since then.

Prologue

At the dawn of the company's development, even before the serial production of flat-panel X-ray detectors, a project on a digital camera for radiography was implemented. It was a monstrous thing weighing about 20 kg. It consisted of: a CCD sensor, a homemade USB frame grabber, an optical lens, a phosphor layer on the lens and a mirror prism for removing electronics from the direct X-ray beam. The optics allowed scaling the small working area of ​​the CCD sensor (24×36 mm) by about 10 times, which could already be used for X-ray inspection of large objects (namely, female breasts).

The project did not find wide application, they sold a couple and forgot about it, but left behind a bunch of boards, high-resolution CCD sensors (11-16 MP), scattered docks, firmware code and an application for capturing frames.

Crazy idea

Passing by this pile of high-tech junk, I remembered that this kind of X-ray camera with the same 24×36 mm sensor (without lens) is widely used for microtomography. We already had developments on our own tomograph with a 114×145 mm detector (50 µm pixel) and discussed design options.

Someone else's, freebie – take it, take it

The efficient manager inside my head carried out this command, pulled all the boxes out of the warehouse and started thinking about how to create a new version of the detector (camera) on a 24×36 mm CCD sensor with a 7–9 µm pixel. Fuel was added to this fire by the knowledge that we had 10 decommissioned CCD sensors left and this is the most expensive element of such a camera.

What could go wrong? There is a working set of ready-made boards for reading data from the sensor, source code and assemblies of the application for capture, abandoned sensors… We just throw away the lens, slap phosphor on the sensor, develop a new case and voila – profit.

Part 1. Well forgotten old

Problem number 1 – direct access to the sensor silicon

You can't just throw away the lens. It focuses the light emitted by the phosphor onto the photosensor, which is usually protected by glass. If we remove the lens, then simply putting the phosphor on the sensor glass is not enough, the emitted light will dissipate – we will get defocusing/blurring of the image. The phosphor must contact the photosensor directly or through a layer of optical fiber. The second option is preferable, since the optical fiber also protects the sensor from the effects of X-ray radiation. This is a common world practice for such devices.

In general, you need to remove the protective glass and not damage the photosensor, but there is a problem – the glass is mounted on military-grade epoxy and there is 1 mm of space between the silicon and the glass. Such sensors have an option to order without fixing the glass with glue – but then you would have to spend money on new sensors, and the whole idea would lose its appeal.

I'm a fan of Conan the Barbarian – the mechanical method of removing the glass was used first. Epic fail.

Mmm… carefully finish with a file, we can do that. Lots of glass chips, torn crystal weld, damaged silicon. Destroy achievement achieved.

Next, the thermal method is heating the glass for local delamination. Fail.

The number of sensors has been reduced by 3 pieces.

Thanks to my colleague, he remembered his old work and femtosecond lasers. You can use a laser evaporate epoxy under the glass and it will come off by itself.

I take the sensors and go to Troitsk. We prepare a test stand, go over the adhesive connection of the glass with the sensor – a miracle, it really works. The epoxy evaporated and filled the internal space of the sensor with gas. So we lost another sensor, thoroughly polluting it with carbon. We started thinking further.

We developed a cutting technique, first using a thin beam to create exit points for the gas, then cutting along the perimeter starting from the outside so that the gas/carbon deposits do not get on the silicon. We fixed the glass with tape to relieve stress and prevent it from cracking during local heating.

Result: 6 sensors left, but all with the protective glass removed. The project price has increased by +80k for laser services.

Thoughts: All the details of the technological process need to be thought out in advance, and the labor intensity of production needs to be assessed. At a minimum, brainstorm among colleagues, without including Conan the Barbarian. Perhaps someone has already solved your problem more beautifully. The path to Troitsk would then have been the first step and 3 sensors would have survived.

Problem number 2 – protecting the sensor from radiation

As I said above, it is necessary to install optical fiber on the CCD sensor so that it weakens the X-ray radiation and increases the service life of the device. We already had a procurement channel established, but all global competitors used optical fiber of greater thickness than our standard + purchasing in Europe would take several months and NRE cost. We found the Optik plant in Belarus, which produces optical fiber of the size and characteristics we need (TBF10 material).

Surprisingly, everything went smoothly, the plant worked faster than the standard export procurement channel, and the X-ray attenuation was even better than expected in the X-ray range up to 100 kV.

We made a fluoroplastic fixture for positioning the sensor and optical fiber on optical glue, killing only 1 sensor during the gluing process (we did not take into account the wettability of the aluminum fixture material and the glue flow paths).

Result: 5 sensors and equipment for installing fiber optics for two types of sensors remain. The project price has increased by the cost of fiber optics +150k and the cost of equipment +15k.

Thoughts: Changing what has been proven in the project is a risk. We got away with it, but it is clearly not worth experimenting in such a situation. A good practice would be to use two channels, a standard and a new supplier. The assembly stage and wettability problems could also be foreseen, at a minimum — experiment on “killed” sensors. Do not spare time for debugging stands.

Problem number 3 – permission

Roughly speaking, the final resolution of an X-ray system is a function of the size of the photosensor pixel and the resolution of the phosphor. We cannot influence the sensor and its pixel, the phosphor remains.

There are two main types: gadolinium oxysulfide (GOS, GADOX, “gadolinium”) and cesium iodide (CsI, “cesium”). They differ in structure, see my article Digital X-ray: from alpha to gamma.

Since we need to obtain a system resolution approaching the pixel size, it is necessary to significantly reduce the thickness of the phosphor layer. In addition to the deterioration of sensitivity, a number of other problems arise. Gadolinium is characterized by a grain structure with a size of 5 – 10 μm and this creates areas of strong heterogeneity of the phosphor. There are no sheet phosphors of such thickness (or I could not find them), only direct deposition on optical fiber.

We ordered deposits with the thickness we needed from several European manufacturers, tested the assemblies on two live sensors. Gadolinium gave a worse result than cesium, but this was expected based on its physical properties.

This is an X-ray image of the JIMA standard on the X-ray machine of our friends from St. Petersburg, we managed to catch 15 µm resolution at 60 kV. A good result.

Extravagant options for phosphors such as YAG:Ce were not considered due to the exorbitant price tag of several thousand euros per sample.

Result: 3 empty sensors and 2 with applied phosphors remained. Litobzor and reverse engineering of analogs helped to select the thickness of the phosphor from the first try. Luckily, we reached the limit of 15 microns. The price tag of the project increased for the production of phosphors 1250euro/89k

Thoughts: It is necessary to plan at least 2-3 iterations for selecting technological parameters, which, given the peculiarities of importing non-mass components, can take 8-12 months. We managed to get the required quality the first time with the help of literature analysis. Do not neglect the body of scientific articles.

Problem number 4 – old electronic component base

Live sensors — yes, boards and electronics — yes. We collect samples, but here's the problem — only two board assemblies out of 5 work. The rest are dead (it's impossible to diagnose the cause) or with an incomplete list of components (potentially also dead). After 10 years, no one remembers what's there and how, even with the lead developer nearby.

We have CAD data for the circuits, isn't it possible to order new boards and assemble all the sensors? Of course, we can, but some of the components have not been produced for 7 years. The problem was specifically with the TDA9991HL chip, without which the CCD sensor cannot be started. In three days of searching, I miraculously found a company that specializes in discontinued components, and for a whopping price of $75 per chip, they supplied us with a small reserve for mounting new boards. I wonder how much they cost now, taking into account Covid and 2022…

We had to reorder the boards with customer-supplied raw materials.

Result: the boards were sent to contract manufacturing and assembled using old CAD files. The project price tag increased by the cost of the chips and contract manufacturing. The project price tag increased by the cost of manufacturing 5 sets of boards and chips 135k + 25k

Thoughts: The obsolescence of the component base should have been thought about in advance, the task of reproducing printed circuit boards should have been set at the start of the project. The lifespan of many electronic components is 5-10 years, finding them later is a pain.

Problem number 5 – USB bridge driver and firmware

The outdated component base came back to haunt us once again. The USB2.0 interface chip remembered the times of WinXP and the transition to Vista. It had never heard of Win7, and Win10 was something from the Strugatskys' World of Noon. The driver in the file dump remained only for XP/Vista.

Well, we took the latest version of the driver from the manufacturer's website for Win10, rewrote the inf a bit, disabled driver signing in Win and launched the USB frame grabber.

Yes, the firmware from the file dump was flashed successfully, but the device did not want to accept the new serial number and launch the image capture software. I had to conjure with registers and by trial and error find the magic offset with which the serial number and initialization configuration were correctly written into the device.

Pain, poorly documented debugging tool and a protocol without auditing… mmm I love it. How can you not love such beauty:

Result: new boards were flashed. The software started working. The project price didn't increase

Thoughts: When raising an old project, it is worth considering changing the development software. OSs disappear, support for specific hardware is lost or changes its behavior.

Problem number 6 – someone else's code

We were lucky with the capture software, the person who wrote the application was still in the company – so we simply rebuilt the application with new USB chip libraries with Win10 support.

And here is the firmware of the FPGA on the new boards didn't startHaving the project code didn't help identify the problem. Searching the backlog, experimenting with forks, all to no avail.

At this stage I was already very tired and decided to close the development in the form in which it was. The amount of labor costs for the resuscitation of the FPGA code clearly exceeded the mythical budget of the project in my head.

Result: only 2 boards from the original set from a 10-year-old warehouse worked. All new boards were written off as expenses. The project price tag did not increase at this stage. The total cost of the first part is 494k expenses.

Summary of Part 1: An attempt to revive such an old project was a stupid idea. An old component base, ancient code, lack of understanding of all technological stages of development and production, poor communication with colleagues, an attempt to solve problematic issues at once – all this could have been avoided.

To justify myself, this was the first project of this level, I was young, actively studying, it seemed that everything was solvable.

On the positive side, we managed to develop a technology for applying optical fiber to the sensor and select the necessary luminophores. In a series, the technology would pay for itself quite quickly.

And yes, I sold 1 live set of boards+sensor=case. It was not a good decision, but it allowed to finance the work.

Part 2. New fashion

The mice cried and pricked themselves…

Despite the negative experience of reviving the old project, the idea of ​​getting a new high-resolution detector on a CCD sensor has not gone away. The Russians are not giving up.

External incentives – customers for a raw product/concept started appearing, they promised 100,500 sold cameras and ready markets. The reasoning – imported analogs constantly burn out, they need to be replaced regularly.

Well, okay, if you’ve already signed up for the development, go ahead.

I started smarter – I did a market analysis, looked at competitors, chose good-class CCD sensor models that are relevant for 2020. I bought a couple of engineering sensors and started sketching out an electronics scheme for a new camera.

Result: the price tag of the project at this stage increased for the purchase of sensors $3450/ 224k

By the way, it was funny with buying sensors. Classic of large American corporations, calls with 8 people, of which 1-2 speak, questions about thousands of units of consumption and so on. Everything is OK with the economy for freeloaders there.

Arrival of the Black Swan

Our CCD sensor factory was On Semiconductor (formerly Kodak). In early 2020, they make an announcement about closing the line… A month after Skype with their representatives, when we discussed sensor models and procurement. Surprise motherfucker

Consumers have 6 months left to order the required number of sensors and then that's it, look for them in the warehouses of resellers. I have a project that has already started, engineering samples of sensors and a ready-made concept of the camera.

In such conditions, developing your own electronics is already money thrown down the drain. What to do?

Necessity is the mother of invention

As was said above – the blockage is in the electronics. But what if we use a ready-made one?

I started searching for OEM companies that are ready to sell ready-made electronics for the required sensor model. I found some guys in Korolev, we agreed on purchasing a dev kit of boards that we can build into our case and install a modified sensor.

Result: the price tag of the project at this stage increased for the purchase of dev kit 282k

Frantically bought a stock of sensors to somehow provide myself with work for a couple of years ahead. I reworked the tooling on engineering samples of sensors and assembled several samples with optical fiber and phosphor. Launched under X-ray – works, decent resolution, breathed a sigh of relief.

Result: There is a reserve for production. The project price tag at this stage has grown for the purchase of sensors $11,715 + $10,002/ 1,411k

In terms of resolution, it's an excellent result, but the downside is that the CCD sensors are extremely slow. Exposures over 10,000 ms are normal. And overheating, you need to install active cooling/pelte.

Here she is, the transformed heroine of the article:

Summary of Part 2: Despite the surprise from the manufacturer, I was able to quickly switch to a semi-finished solution. Ours in it, however, only the technology for producing X-ray sensors and the housing.

Thoughts: The path through OEM integration should have been chosen first, as it allowed prototyping the product and testing the market with minimal costs + the work would be on a current hardware platform. Dependence on a critical component is a big danger. The age of CCDs was ending, this could have been foreseen and the transition to microCMOS sensors could have been made in advance.

Epilogue

Now for the sad part: there was no demand in the end. The people who promised hundreds of purchases disappeared. What was left was a working product with the price tag of a full-fledged X-ray detector, but with an extremely small scope of scientific application. Fail

Total cost of the project: 494k first stage + 1,917k second stage, total 2,411. Not bad, huh?

Thoughts: Market analysis is decisive. You shouldn't take the word of customers until there is a contract – these are just words. If you are promised mountains of gold on the sale of a product that has not yet been developed, it makes sense to include healthy skepticism and ask “Why wasn't this done earlier?” At the very least, double-check the information from another source, at least open data on government procurement.

I hope you found it interesting. Learn from others' mistakes 😉