we sew in parasitic traffic by hook or by crook

State censorship is a multifaceted monster that closes the way not only to foreign information through blocking resources and their bypass methods, represented by Proxy, VPN, Tor, but also constantly trying suppress uncontrolled secure and anonymous communications within oneself. For this purpose, government agencies are ready to resort to literally any tricks and intimidation, including fines (Threema case, failure to transmit encryption keys), criminal prosecutions (Dmitry Bogatov case, VIPole case), regulation of uncontrolled cryptography (history of bans until 2016prohibition of encryption of domain names), as well as possible backdoors in their own cryptographic algorithms (Grassnechik and Striborg). Racket becomes the norm of life for the state.

At the same time, government agencies are not embarrassed by the real lack of security right under their noses in the face repeated And massive leaks of personal data from Yandex, Delivery Club, Sberbank, Alfa Bank, Rostelecom, Russian Post and many other companies operating in the Russian market (list1, list2, list3). Given the existence Yarovaya package, which collects huge amounts of data, such cases should be the exception rather than the rule. Such a law was created not only for the sake of combating terrorism, it's true? But we see an obvious contradiction, where, on the one hand, increasingly repressive measures are being introduced against citizens in order to protect them (apparently /s), on the other hand, their real defense is constantly putting everything under bOThe main question is reality itself, as well as the security measures imposed. And let’s be honest, the Yarovaya package itself leads to the previously known concentration and centralization of personal data in a limited number of places, without excluding replication, which also calls into question the rationality of such security.

Due to all this influx of news, developers of client-secure applications and anonymous networks are given additional tasks with an asteriskwhich consist not only in the implementation of persistent applications to the actions of all kinds of attackers, but also in the implementation hiding mechanisms the very fact of execution, both in order to prevent blocking and to hide obvious activity from the state. Thus, steganography (as the science/art of hiding the very fact of the existence of information) becomes the last line of defense cryptography.

Goals and objectives

Our main task will be implantation anonymized / spurious traffic to a centralized service for the purpose of subsequently hiding it as normal HTTPS traffic. At the same time, complicating the situation for ourselves and making it more interesting at the same time, we will try to use third party server, not owned by us. Such a condition will help not only to complicate the environment in which the anonymous network will be deployed, but will also make it possible to better understand the very specifics of such “implantation” in real conditions, where one of the important points will be adaptation our network as a parasite on the body of an unsuspecting centralized server.

Also an interesting feature of this scenario will be the peculiar bypass NAT for P2P architecture, no matter how funny it may sound. In this case, the server will only become a relay (TURN server) of messages from the sender to all recipients, without the ability to decrypt anonymized traffic. The advantages of this approach lie in the lack of investment in renting your own VPS relays or your own hardware capacity.

Unfortunately, not every anonymous network is suitable for a task of this kind, because as such it must be able to abstract from the communication medium itself. In other words, the level of anonymity of such a system should not depend: neither on the number of nodes in the network, nor on their location, nor on their connections, nor on the level of centralization. Popular anonymous networks such as Tor or I2P become unsuitable for this unique task simply because they are unable to withstand global observer.

Unfortunately, also not all are theoretically provable Anonymous networks that can resist the global observer have the property of abstraction. For example, DC networksin their classical execution, have a clearly defined connection all-to-all, where the possibility of creating any intermediate nodes becomes unforeseen. In any case, DC networks can still be upgraded and brought into the form of abstract anonymous networks, but in this case you will have to add end-to-end encryption (E2EE), and also, due to the lack of live projects, simultaneously write an anonymization mechanism from scratch. Therefore, the most rational solution for us would be to use a ready-made anonymous network, which would also be abstract. Fortunately or unfortunately, you don’t really have to choose, because for such specific requirements, at the moment, there is only a network Hidden Lake.

Hidden Lake and QB networks

The anonymous network Hidden Lake (HL) refers to QB networks, in other words, its anonymization algorithm belongs to a queue-based problem. The essence of this problem can be described extremely simply by the following algorithm:

1. Each message is encrypted with the recipient's key,

2. The message is sent within the period = T to all network participants,

3. Period T one participant independent of periods T₁T₂…, T_n other participants,

4. If for a period T the message does not exist, then a false message is sent to the network without a recipient,

5. Each participant tries to decrypt the message he received from the network.

Thus, from the perspective of a global observer we will only see fact of message generation during certain periods of time = T, without identifying cases of sending / receiving messages, or banal network inaction. At the same time, we will not be able to separate false traffic from true traffic, because QB networks, unlike other anonymous networks, don't impose false traffic to true traffic, and replace false traffic becomes true.

In addition to all this, the static periods T themselves are not a prerequisite for the existence of anonymity in QB networks. And indeed, if we make periods T random magnitude, then this condition will not change the essence of the matter. Due to this feature, anonymous networks based on QB networks can be more flexible in the tasks of hiding traffic than their counterparts in the form of DC networks.

The Hidden Lake network has a decent number of settings for all kinds of adaptations to various network environment conditions. HL can customize size generated message, the number of randomly added false byte, stretch periods and do them dynamic. All messages generated by the HL network have the form random byteswhich to a certain extent reduces the risks of unambiguous detection of the network based on the data structure and, as a result, reduces the risks of successful blocking.

In addition to everything described above, in the Hidden Lake source code there is such an entity as – adapters. Their main role is precisely to adapt to a specific system with the further ability to forward and receive anonymized traffic. In a way, the HL network itself, at the level of its source code, becomes a framework for the development of such adapters.

You can read more about abstract anonymous networks, and in particular about QB networks here. You can learn more about the theory of anonymous networks here And Here. You can read more about the Hidden Lake network Here.

Centralized service

Based theories abstract anonymous networks, purely technically, we can choose absolutely any centralized service in which it is possible to write group messages and read them accordingly. But unfortunately for us, some servers create additional inconvenience when automatically executing scripts, such as – registration, authorization, captcha, bans etc. As a result, it is necessary to choose a centralized service that would not be so intrusive to automation and would have the possibility of group communication. Anonymous group chats or anonymous threads work well for such scenarios.

One of these services with a fairly low level of bans and the existence of anonymous group threads is the site chatingar.com. He has the ability to both write in group chats and write in threads. Group chats are limited in number and cannot be created, which cannot be said about threads. Thus, in order not to produce unnecessary spam obviously, so as not to spoil the lives of other people in communication, and also to be less noticeable, we can write either in a long-forgotten and old thread, the activity of which will only be seen by the admins themselves, or create a new thread and write in it already.

Also, another possible candidate for choosing a victim was telegram. It has a fairly good API, which could make it possible to quickly deploy a group channel through which all anonymized traffic would be sent. However, the choice fell on chatingar due to a more fundamental understanding of how adapters in HL would work for an environment that does not have explicit documentation. If you are interested in implementing some kind of adapter yourself after reading this article, then as a “homework“Telegram may well come into play. Plus, you can send a Pull Request to add it herethereby additionally participating in the open-source development of an anonymous network.

Sniffing out the API

First of all, we will need to try to get the contents of the thread itself, namely – number of comments in this thread. This condition will help us unambiguously determine in which specific place the generation of old traffic ended, so as not to pull up all previously generated messages when the application starts.

In browser requests you can see a request like https://api.chatingar.com/api/post/65f121625b65dcbdedcbed7dwhere the line 65f121625b65dcbdedcbed7d – this is the ID of the post, which will be very convenient when specifically indicating the location of generation of all subsequent traffic.

The request itself is extremely simple; it does not contain any identifiers from the site itself. The only remaining identifiers are the classic ones for us, namely: User-Agent And our IP. The first one will be easy to change in our code, the second one will be easy to bypass using public proxies, including Russian ones. However, in order to simplify the code, I will leave these parameters unchanged.

And in the answer we really expect the number of comments in the field commentCount.

Next, the next thing we need to do is to find a request for getting ourselves comments. Fortunately, it is similar to the old request, it just added an additional argument ?per_page=5 and instead of the line post – line comment. Full URL: https://api.chatingar.com/api/comment/65f121625b65dcbdedcbed7d?per_page=5.

We received a very relevant response, in which, in addition to the message itself (body) there is also a time when it was received by the server (timestamp). This will be useful in remembering previously read messages.

Now we need to check how the API behaves when the number of comments is more than five.

And the API behaved doubtfully, doubling the parameter each time per_page when loading a new page. For consuming anonymized traffic, this is an extremely bad property, because it will constantly consume more and more resources of the server itself. If you continue like this, the server will begin more active attempts to quickly ban this kind of parasitic traffic.

And here intuition quite gives the answer – what if, in addition to per_page there is also an indication page? Let's check.

Happened. Thus, we can ignore the parameter altogether per_page (by default, the site always sets the value equal to five) and use mainly the parameter in your implementation page.

Well, the last thing we need to do is check the API to create a new comment. Fortunately, this request is extremely simple and simply boils down to a method POST to the address https://api.chatingar.com/api/comment indicating the message (body) and post (postId).

One of the important conditions here is also input verification big line text, and this is where we are interested in the action of the service itself – whether it will cut the entered message into several small ones (as VK and Telegram do, for example), or fill it completely with one message. For this, I prepared quite a large string, which is one message from the Hidden Lake network (8KiB) encoded in HEX (16KiB).

4a61831da281bbd06c010fb02ad9b79f64fd47a01377c4805427658f4f6dab3415a3bbb0a3fe0983e11d8c6a01700ff45ebfed9acea4129c966db5e82d0e2ae07c78cc8fc164d5172cf9faaca4e869a761f2706f24c355833da1a25adbb6da2fd91fe82f9a433803134071a965f39abbe6f8540bf39a5cadcfba50c719bd50a09af74abacc4dcdbd32fd71e85f7573742e30febf934b533debd109d014caace9e9bf06f51935cf7b2e64ffcb867b47b3ca02a5ae48a864a7ddc255531193c323f8b616096cb33486bb9a1df73c412e19d7f21fc4adddbc1b7996e31cd24ac3ec0309dfb9486154d3143d189f0b01af7ecdd2417dfb10e6226e247fdc67a8fe0ba74733649993fec335f91c68b8f7f4b5fd88f078fd0503da282d6888ddedebdd609cf0e6a1305b31d3f51c325e6da2276b20ec04c45167ebefe8c9d8b446d662bd262500e0a5cc1dddfb654ce0bc644ca381b068c20f7773ed3d0ffeb457377f7bd2858e7d0b57b0998a0a58a3c3b571d324fff3d67182ca4762b193d596a29eba95aa5154bec9fc0d1b4ff6d5cd367dcc9d8757463fdd5475c068a77e6da8ce3db4227e39657838a6e67a7b1ad139c9ae231a7e8bd90f16ae548fed8e516632907bc9fd9937d176318b24ed45964914fe5d634bef1a1c8b69ff517f9e65bac8a22cfd4acf57c46d8e22bf5eec8cf037b74afaecf317a2cb0f965b0c654e2a69dd6729cdd7d4d9b094d3a2c24a1b86578d8d6e86bc656d6f0bfb9575e8523fc83720658c341f24016ec4872a3d04b368a7e990b55ac171e147fed4a1964c590cb1f814507e49af4f0e35384f4161f9ab56b13bd0b902ef5f61c6ea46128998a835fb923fe552bdc3a0640b48cda6c61ba984583c4ea1090399fdbaa1d9bedca18b50605deac3632b7cec8a0ebae6845d0c37a0e7c0ce411c601ff3f6e4da0b4bdefbaff4775432ff7b024ba47bba110e366a7a89390079ccfb0990300121ec05ffbb9907f6347bafe675c6f10e5f468e74730c93634003c20fe63c112d5f4aac52ded926c181b6965c7801b9b3107126096afe53256b69ecb3855f13e55c3221d2f85ba74f90b0f29727048802f6c215418209d66abbd8c615f298df130c6e0bdc514c85a77b6fd35bcf65aa9c0f18550db732d34755fbda752985e566772e24b308d19984a1cf2c4b614c15016182691a0b76e43490adb0ce6a0bdd2718051b46ba7b5a88d606f4c72e806a45691db9821164141419514bd2cc91a5ac8f40f906f88a3db0597d90e067e754a4c005a76e54c112d83805b53a07128723fd3bfee2d01fc89cfbd90f4677bbdacecd30e94589de0e7261c70f88631e306c1abb156e97f80f9f5cb1511855afd00e91f2aa918ddfcbbe34c3e8f0474c04924894b856a4b164f0301e6a17a435e06334926f7e8764be2dc42446317fd131678c4c58e9cf155a63bdee03916ee327d49270bbe4809d3d01da62bae4f37c548708af216f107f446ff8be558252e81c922b8bbf45e095efeb87c15bc52bb47d10cc116b04ae52bcfc0c432beff4430255dc601ccf17b0da32d978e5e6ba6d9b5614f445a8577f3c55369a500211097b073ea376f28191923aacfe5dbd72c1e9b60031622bfa8e56c86d039028e1c690e32d4efd18078e53955ba67c6b51e55cd7f70ca0359f02b7fb1da164f2bc1aa30e918724de7e72babfcb899b4a360fbcd85b9e4f312b02f8a8ca7497f91b01727e9701a91e97e629583427610448c6a82c9a7f1b865eeaf6a35b57f10f88ad1ce2032a4a2242ac77c24df98001cb7322a081cfb01b602bccbb6f7856f4fbd5a8e1423908391da4b18c749e9869c94a7307c0cf1c726f0a650531171bb782a747752e8c0fd0153d60b6ded5a0427f739e16ed21b18d51a78b26d941ae1c748da8b27fe5d1c54c4e5d04c14e092d96ccd03ecd2e7c54c0c61159bf7c5d3eaab19ba6987f63ff0654b6a8b8ae7f9cd08693ac7c902a6062405187988edb38b6da5d8ce921354d99eb59d55da1830c02bc95d5c90152c211e48bf562d72e46139bf73b11c2d2e6841053a86f45dadfe8017d186ba01bfa8dcbddd9af05ed3d6349b7d666e347fbc2116f187084d93164a731ca503baf7f31f579a7d390194bf3d1e61b5fa38da3bacb5f1ae6e7b393f0f68d80132cdb747ec89260d2a9ea6b3e32127430a93d2d99bc670ba84cfd9275fd14d70938221a7cf62902fb0bbcfd786f9c6ec13efe0a88d8356f4b4f5a263c41168e5040de34bc95984ff29db100abf9dd488c8132010f0bd0223038dbccc179e636376b8102dc8d379e1abba993e40cb41e46223b9cb67c3c278bd730dfc1e251bc80e90160b854026b85bbc4a2835d9505aac055f33d8c47883b9980ab49a4326f423d95c0bd2e9601645a053dd6aae059025924559b9ca871e4c123fb6a3a0385c2c631d0f5c8de4c55d9cf76041a4eedf08be76742584f726faa60a31a6ff2f9450565a3a65d4e07d005bae58af03ebde0a7fa0ca3ef1647043055f31e03be40415257e2fcbe9220cd5ce381134042e85e5fc70c39e4c05cf2abc4a893e725c71497f942b4901e195401745ce05fe8f1923ad4c7cd7810c601b9a5cb81e726b7cbaadda9b867e0a21386d0e1c3b9571d6a1c3bdac47c2b2640232ad07ae2abfddaa01fd1916699651e20fbaa7f0067befe83dd19600e8a51047075ef2295a68576a7b89d8087b30bc1d5d51258d82115bb7c6fdc673a9308a460212347b85b68e4bd7a5b3d5cb3fe4c55e3136d2b3745a5e65c921805ca58f91930ce8d7953c51b605e7caf684042ffbbefab7fc781ea9ca324864962d419f14c2a5abcbba2842bbdfe8038effefa08add2e398f00935798dfe01c8428c036b8839085c0b447fcb17e160b84048272d9a57da16d3351ea4f4501595566ab62c8d05aed55098d9c388a5ec8d7a7dc42f61f34b02e3a015302faaef930c506d57a4e5c94affdd0db41a8aa5323b8edc9e016287cd246e574e1fa480084a34939e61bcded3d068fb939de7e000f60b0194f6f66b1e08afa99523d7077e3223fe0aafce2428c9ed8dc089514543fcef23872b277753c31d3b3d1c5d06cad81aaff3ed486a3d01a74e86d6ef0b7b3a2e733315366ddc64c81baacd8753b2c3fa2e3b8710491f2d33abbe3d1f515a136def9b5fdd32a2a348106f2f21619c0338f986de968c15287c195aa045d288c983a74de46ed554f41d8b5eb579d24946fe5974627b9cc3c212de6afba6c9762dde9123d8a11e5083140d9401fe06d32e98d2990fb7588cbe4db823bba115da28e1e8be20d5bffc034fb01174206758a8b9e0bd4bd3f18540b47e5a4d014e7083fc73fd91b394203592266580f9179475c937f9edbce7ed1088de6d615af6a9d4af1c940ab5ff223e4c6adac7970083e9bbe30b7b350288417d4b80f86d3a955e1b8aef19eecc1dd717d80b03d75bf972352309474ad3ec47bec793691565ba3dec64a75e6e15a1bbe1b83d7a8d0e53d655d7226804791867c210db8d9762d20d4fd1d0ed268c0354ee300092e39250f6d31616439fc63219aaeff72db13c7a01852931dd7eeec007e58b11ebcbfb23c4cd8ca7772cbb463ab86740a8577b89b67cdeb2035e8cc24c01e96e170867f05b9a10beb10760549791660443a9cd7df0dffdb5842f5426b566e3dd7fb5080ebe171934381081ab710c091a0695f56d70386c7ef0c611143bf7ffbb9318ed204ddeb3b13cd846cbdf3702090147d371ead3e99f21c59649beb9e683dcd0fc1f10d98b8f3e8511309f25927f11c84f0ea0b55a1b80141ec8d64b48acdf26a89ff4c6c692df2d42682a795edfd88ee0af5fea8f4bb85e4ae4984cfae51c38e888ddfb616ca04526ac72b19deba3c3d8f516b670512e7dce82cdd6be463fba91e63a4599cef0bbe419d6530e3735779e9f3bd6935111a4b667f78a3ddd1c3b79ea34f1679de1f820e0772959c84c0b179a5ed618daf0270b7bd9300ead11e1bd38e7a42ba62ce329cd4e80b5792f4dfe7e804a488ec23f88b3a45cc5332c9bbe37800882a5cd0d00278c8972d02425e9799a497d0330050d757f26f8262b531cd284769b10a8f9d675fe73f6f3f30f7aaa612a223118bf03e919e61a9121c336a2ba661d9c7ba02bd8721550d7aa2718d0cbbf39dd284b5ab370282cbd191831e20f91c1d6b5ec786dac2332bf802681231522f303c0f1b646d4e45a53ff90778193a3edd6408ad9b89b808276612cdcc83ed4f0beb74fe585f3aaa42363fbb4d92ec5bb8420a9f11a0b64d743498b2c7a62a52738b17a61917aeb2e28dbf1b406ab5fcbb1bc74f9a9e961fc3f061b96f3e0ee794e0e796d97f051340d03c0145a6576a525fcc47decaeb3a13426abc7bdf036e92ba072d0c88831cb823babe27d80f3cec6134616593e7446a6af2ff65c989a43476f9bdf2e2bc98fb78392f47e15f21fcc3730e1db4803cc3de617980b35ded6874170b2e56f86f36e1797dc42f72ea192d62654f3982e2637253afe5196805e7f0098f851b5c08a03ddf682c5937ffe483daa130be95ebf0214472fcf4d2f06ff90c64ac109de91bd7768404c087fe21ffcd69bed5559482a9228944792ea205069e4eb2e15e555d740b95d221c9064557653ecfc6251806831345f166bcb477081c3492d4071fb740030b88f07b0e1e00ebda772727ff092ccceca23b491ed4663de35ac826ab522babe371c07d7171fc81d2a47af2af6d7906ea0ae462ee9e343939796c60abb48fa46389fc83b78f33417927c5444c45c47309d03a2ce5f27e6394250242cdd80fedc61c999f4051c2457fec2fff92eae7df1731a39d2362f3ddc47f2827d3c22f0f91fd4445de3aa1c66c962352f7d591f889e15af04f5509f4a1df0eca5458749e36befffc5913bf6f2ed4f213642cded3dbfdca3a4c3a19383f772b5ff97c889faaee819bdfaf1bbaf63c3867808babb46f6cf3146d71fa901cc38a4ee675ec4fe871971ede532906e55e025c45459b91725230930eb2529b053169398dcfe0b362d554b5443f888065fbe2345b26828a96b78136c644334817b231c6bc31a9c4039538f348a667285be85193e904c8637bcc70bb41538abbba87473de9092f2061f6675488cf9a41ce14501e08df6fc50be729172a75f0969671bda19c350392a172aa869704f0ea2cad9caad0437a21b606a669641b8409417ff1554833ef956f15d3ca02f9b808e04fb6920df4295fae5f8a9b59d854a70f9a4638c1143639d9e8d56802ada8001257885e2ad56b0a0dc403e0c6c7a415c2a34e3eec3313d0e179a72c4bf326f1013c0d63ccd275f98b59d14a6fd079b553a39bb09bafdab3c34f4f6e345961114f92332809c465167ef74a9259a3959a3130147d605b4e086e9ff5bb4c87c6fd7e185c9ca23bdc6130d3ad15a9ae75d2a1daaae84273209ae53f3795eca5199a91646a67ee2ca10b5cc2845324aa120e0d21164d39296b6a039e0174da9a0ccc0ce4070058363a88bc511419d6d82a6ecb1c04d753b57a7aa19d645c468de7dddc4daec308ce571454827d218ade8bfb338c0a73d9f9c6e0de8d81a14699162da41aa6ec219e073e2222558ad0e9a6bc263fe14137d4be2719a3770033143d62a898e113aab37db26389a80474a1a32038ca530ee773936de582c29d1a0f0ea29d16799a34fdab620042cbedf6e4215c53486bf096dfb289b7474944b5c14a2802105dfdbbd893298cfc15fedc25f1b92bfb24516d12039b9105cb4c487fe3fca717d05990ab46689ca080df9eabb562009ae6fe5bb4bf9f31389f0e9a86e185b9fa4ced6500491c569ecde12bedf7c2a0f57283d3c69077bb0dd598d77a48f3297eeadd1d95f7d31c326451937614e6c1fd82073b8e64449637f84eb3743243fa382dd6e4201c147390b66fc0ebb0a802f6dddacd673e7f6faf7be2bf92f0d995ade09c6c5677734bdaf94a2f18bf122227badd7eaa49e7e28ef97c6a46be270e2da038445f8b39bd4274b679bb14b915334f46622682455746369a18d594fa1e5eb6fc31fe5102574c76b1f110211cb33d63ddee0988e5d10afa1fa11972dbd07f383b3b09e2d518ad7aaf05ab7031f45ad84b69178ee02fe2139a2d4247034624636388e3c2a0522d77004e50f1c9e7a025e470c15572aadaac016fca9ba23ed5243c041840ed0b1536f3b713476df289776c88aa749d78fee99f10fae68b72b27e2c6a62fd528a5f95e153635fc521085923c6651e2d3ab27a1b733d7d7cd7684317f262cb509c27f11f3d5eb8b83a71ea16e88ff4d597cd7cb7fee47c7d1bd1037b5763f569fa57af84e032a3d2c600ee15522ef735977b5cd6a5811f7c64362c13d7376cd7bba7db6204ace2ffe0f94e7bfd77ed3150f03252b86b49874b026a90130a8cb75c8ce3699a7f8f35941d57af55b27ee22a05e385f1ae5926bd64207dd933aa941db8f512d08fd51971a6a4f96566b36ac8127c00cc110cbdf534ec6cb6e524b1f9d9b762bb24d9beaf520dfd6cd224f857ab46bd0ef939689d1137e0d88a434ac7ec8930e112eb06d420f831defdd0789531ebea4eb4c031d6149628c33d580ab2a7f87a015c7f779367c643a463c69cd4751ddb5e3d091cd4e15ddcefadbe1e3e2175f3a602d455c8aca59f8c60a5cd83787b1ab7ecd318f41ad3281624092e6ba7fe79e0fccd1047b164fa70c72d6ba930999878b7201554b30b88becb84e9be255c8832699675c2156bd334d95ca400c2038f8622d76c820ea644ba50c99f6e6a817f4d2d21ac40d83837ca9fde6d979e36b980d4d74a4e158c526b4e189087af5b011a73c9da4368a3e92d870a9f4a0ddc64a572733fd4b50d6313daebd70e52b5f05af4a543d61f91442ed9e9e70581428b9c27c684feb58820edd96285615efd2318973b28a5f01742391dead72cc8fe2acddb4b288ef64f6beed4d3eeba08ccd8d5c199b7dd6b23427b04899b64cbc17cd570cbe8d54bed8585cc2a9d3e8aaece1e796e0406d7f1aa827904a346782298a35e40833c31bd42a511257821a657c966f6c0024e5680c13df978da334b0a8830f1506b1b26c59c185e16466ffa13648e44696e4e6fdbdf6d76b750c4815f9e9d691cfb6c25e8bacb52146a0242a0f3fc84c2852c4e60d92c3d323f9594e7538170203eb5409251864bd4076fa59f162b58574ed004233679589f7cefadfaf3d93367008e3820da2b0867627f6d3949d76e0d90fe2f3f580794285745f5cb40740d27522cb593df780f666a9e2495d44fe436ae412476017d61771a496e21e9c3c6a00f4af88f77f5f38805bdda7e2de89265bd965ce040b4d654cfcc2066c40fa19f62fd7126986d7ffc39e29523e84fd5b36a65cdad5cd3604812df69a12dd8529084a87d5f8c7196999284a5ef59be3c539f2f15f7e4441c0c4aafdde0238bfb9c2c41bbaeae5efe595d4bc248c462c8c3edf2b62d55c4356d25c22d34c3658280675101e0ac0e149f781f7c1016e7cba49d62dbde4262d14ed5002329b6aaee8ba17f432b300227ae0b32cc35ceb030d17db40f300305aa6c8f00b32532356daf9c6374ff712dcdcc800c4277571b63f63605e5fbf54e97ade9348e710bd6d976365dec934ea3e502931649d5a6cbd0fa41a183ec9d8d8ba2b959cb670d1aae57d3212d0f6b5c6277d6984351f047f7e63ed0998af5d29df4ac4567047deed8a816cf6c3ddbf1b7d0a6c23536cf5cfd76c86fd61501eb5f0a779d590d4024fb20d001995b365884ba600cfa9c878ddcfd23f85ae81fd118d954096c7b92d6d25da1f8a74987d04c6ba8adf81239d1ada19aacf6d02c0b37f949b08b404bc309b95ca0f9b587e750d73b78393bae0fd9a24fba45aa3531fb9036ec0e08cc9171e7ef4676be82714182d9969d2851782295d84d2c1c3f978281877d5cdcb92f2c452852ad6794111e9c71c1314396d5c606d5f1b23203a0282f4e86565193295e1aa54f89203952e6fb4c4177cb63c77dc34baeb176e7cedb823d11671e076cf194dcd68ded0ccff82061c4b30198cd8ea928028c05d24a8bf509d1a1fe91c3a1bf55463a581fdec1bfcad07d2e5a6775459cfea7d281b9e405ab532c315a36cb84d23ab6120633ffeee8a7c2cd0c8ddfede3e10e057f1c4e71b483a55271828aeae972e238a3eb86a5e44c20823e978bf080d448efcf6c0380a8a69a5b33e300cca86504d5f96d972bad808d658dfec9cdc18f9b2d1f684cb0f7c6d1296073b12f507c71c15056ef9cfe86cb7e18515ef1179d63038851f450a40b5744193a559a5dbd2eb2e0c887d14e03a3971f07a966b2fa2dcd1ef41c60d908bcc5e073a2b038b69232c047e5a3b33414ce8eefc73652c8f8e791bbacd9a0f010a457bae5700edf5bb25ffd3743495ccffa711db346d2a1029021fa120a6c5b4915ce9082496c56e711dd59d8957bde80904f03638286807dd7d5ed64b8fd72284ebdd12cfa7a409ba9549685680b29d1db8c85c2c1eb4d3c202841bccb2d77e7e98ecd0b6cb41b2e4c83cc31558b319263943ffd4d7e9e6e5c39bcb42fe12ea3285e2723ccc3afd22d7b532576df62bfbc7451c8d6af71d7dff0a11c412c7f4cf2da8099351121ced3ef9ec1020aee055afe0ab769bb948199b8e13760f94b043506eeb5c3c9fff6e4b0fca9c2e9bf10ee19dc26340ff9153c9d9935f1ee54cd3dc2a21537239bec32e44e6efe713dcb643759258622ebd34d08700c49525688ad39c68636bbbd63628dc0bca6bb5bc68159237e1ff5cb7b7ca7608461027d23b632902b3e2defe10b36a2681b523fe88a0391843f633239c27657e0cedefb43aca7b2a783525c688bd3969d05ad82ae6c843de41a3baa187e0e4623d3b66ad76f370b0caaf099717bfdf9c8050504a46928e29af80ba549b1cbd605ecf54bc0e7a529082b9f3fb1e83c86c3945f4efb7f9a599f49e6e4420b675187838dde61eb05af97d6450298a6674bcb275a954b1df14c4792e5a81983e26c14de66fa34b6f1ec195251fc8c8343ea202696e4314c35023dca2d605245744f062a32d39747d8aa7060aefdb1fae22c839ef19a4a9624123901c91ad411c4c91f0ac653181950697e1051b5175b2515963aa5b40b5975b8fef48e8bde55822027fc2c70a1d448dcd1e8feff02dd3ab575f5263882d8fb88bb199840f14f375e3b715a4d2bedbc47de245a46b0284e85c0c0c3c07b2ee6f9886e1f89d043be1d89d7e9c82e8dad9813a6d5f8bd2256ee48d64dd3b04451a3a145f5f4cb445c4207fe8805c79561615d85e691c97b81d445c170943b2b0b5b846adbc2467abc70279b9b1a5465ccba6f2f22d9496ea65c4a40b122e1c704e5dfa01ddbcec59d25d19a400ee530d2218d1ac3213e4a0cf9e54797f015c1d56f1e09e113823663334c361a77c444adb367dcd77c9b64401b27a4fa87c5e1ffec6720630e77a88220fc19884e847d226753e0dd88e652a5ee0c4dc513eea5daf5b447f133d88407e1d8d72c3c166f57ea996110f5db07ac7a904d6c2a79c30dd18f350483d575522753f782c6d6c21f64763cdc0c38fddffac6ed1d66e8ffa28f94d4e8d55be7b6e75f1ad366ace514bba6307c3d8a9e6da98d066d7675c201457ce8fa8246af729e0c1bcc525ce5c1d895f86d72b29701b1c43ab33de5fe1641f9ed99281ce87feec4fd3eb41c1bd7856a5b36eecddb4aa0a419187987dbc2f953c0546a30762f2351dc1ec89a03657acb2bec039f0432e6d9a59a02f051f9b1bd828e5e0ae1e62219d5faaa809dab33b741e311f78464428557a63cf5673e55b70e4157ca8c1b1f1568cca51d203c2a49c547eb049def16cf0f3c6c9f5ff2af676efb1cdcd3951a93e16197bb3a5b403a9e18648be76deb3d25473fdab022cb0abeed6d42debf654c95304a344f1ecded02221a7645dfcbbf3da6ef66eb7ab24f0e7f5c399b3bd1a473e0fc053c254c69df539de7ba0d30cfe230845d3229f543840294a92af12f1d5e5d7d3920231caa0e9fc5e76072d17774d4ced330cbd3ed9a98c4f9443db703c5e5285ffe36e3ea63084198660101af6ba59e59c627ed3e1299a8b1b7a07da263e9b567ca5599c93786837444c4aee2f816057f7e5fc78368a0ce309236a59b6b9774726e6cfeff3778d447451d32a003fa128a18f4b3a64c5c99123fc4988bf4afc0878357d902e4a748b49ebfd50b34232dd650e5c90a43fc8f82edb821345e8ab7a49f864abab385723ca82fd012681337407b78d354a1f541cbd5f90c252b0dc54bc2aea1af5fe614deb093fb6aba2538882f8058c312f0d4102041fc63e8cd57a860fd07558611f78286f3b11f1ff46f77b7a257e430acabe5a1c2382d4daa1e7bbcd52082490fd92f62a0ea4223f33ecd2920c7f8b0a42500e9a09f9b4ce23696487bee94e275f7d635ddff157d81a0eecc9ceb262b7aace20b8f8b5101c4ed45bfb944936367d50f4e747f78fe30fc7f9cb7061e2a92221cc60622b90bb62213d88a956f5e2320bcad0a7b6e6bfd30a12edbd2250a9b9ea9e6b41ace8d2882c7c68683dd0ea77960c177bae9d3e3b54a8ff366132205b749ba76d8667b79c567421e878a512c8a451afa40666221ea6714c064135cf3d1f6a8a28ac10bf4b9ce8d5b0b1a926c80732592bbb22cac92cf107d4491b39274b79f08bba10bfda91199bf1325c6e13f56c468e958d28b438585b062a0b47aed99903ef596f78650175a05b90b3b5f08315e4e662ed4e98b8a1f04945bf566f6747ec1a2399e147f999a41fa846e8fa523e7522a36f1f08f6f31c52c59e68a1d8ae92b37bd63a3d8056167c601e2403cbc58f38c5db53527519615ab9c960ec12875abe5c5c28d36ac2fb47c1a3d7f94e147a66fb5711e49f56e89328ae45384317db64354b53e1c7ec74047a57f3247939ff1b8cf49793f8d5c3501ca98763e34f262ae62f94cab0d43671c96421c0d858acb880632e9cfd64a8b429ab2531d81706be6163269dff96fde9a3d6df212cc12fe57f051ce0b813f03ab3d597866eebb64695e05047926c2bc5e1282188bcc9f624cadfda328ce8d8a538045bac6a31ac4a5e238e79088b96f2aa72139b6bd3a6367b0f9e989886abb1b04120e39b446ab7a52ba65f39ebd1be66d1be8306c741a9c8017a0e86a3449191e23f27cf7869380dc4fbef476f8e8195e1ac7442df0d542fc038cfec11ae7887e651154114554250d8f15ecdc4b25d31f675d9429f0039d9d81bb82ea34dc2cc3e5ad90eb79c194c4c2c7bd95544e57ed8ee0aacaa8739d5a33704f2af29628abf8335a547efccb988b946c47a9df89ca15e1abf895d09ccb48cfbb06b45019ab8b700320c6c933c2068f147bd532aa289b04ad61a827676461f5f8794024d09875b0ee6b9cac9b6cafdad50fc2548e425e799780a3eb10bd2ca74e4731c4fd6e3adf9ee0736e5efc6611b2c290f11ce5c0117672f605a9a54f33ed36221ce15f02f25f5f8110ca35acc6e0ba460646574724ff5e83971785cd99a76ca86da77753d8a6f494cace4de49d2a3fe0467bd6193447c47fe508774dfbd4c14e6cdf866e66b84ca59e227722501acc2bbc215da4298aef93cbcbfb303d0c31313328d9c7af3554a8803ce5bf7798de75d6ebf8a7900c38ae4fe1aca1a40c24df7ba3a1a78a3c04270805da268673414e8a9ecbc71d42ee21fbd929b466bde7569126f0aebb4

After inserting this message into the comment creation API, the message was successfully and completely inserted. This means that it makes life easier for us when reading new messages, without any need to add logic to combine multiple parts of a single message.

Well, that’s all, this is more than enough for us to start writing adapters for this service with all comfort.

Adaptation of adapters

Adapters in the Hidden Lake network, for their successful implementation, must adhere to two interfaces: IAdaptedConsumer And IAdaptedProducer. The former must be able to receive new messages from the network, while the latter must be able to create new messages on the network.

type IAdaptedConsumer interface {
	Consume(context.Context) (net_message.IMessage, error)
}

type IAdaptedProducer interface {
	Produce(context.Context, net_message.IMessage) error
}

All basic requests in chatingar are carried out for the most part with the same HTTP headers. Therefore, in order not to stand out too much, it is better to insert such headers with each new request.

func EnrichRequest(pReq *http.Request) *http.Request {
	pReq.Header.Set("User-Agent", "Mozilla/5.0 (X11; Linux x86_64; rv:123.0) Gecko/20100101 Firefox/123.0")
	pReq.Header.Set("Accept", "*/*")
	pReq.Header.Set("Accept-Language", "en-US,en;q=0.5")
	pReq.Header.Set("Accept-Encoding", "gzip, deflate, br")
	pReq.Header.Set("Referer", "https://chatingar.com/")
	pReq.Header["content-type"] = []string{"application/json"}
	pReq.Header.Set("Origin", "https://chatingar.com")
	pReq.Header.Set("Connection", "keep-alive")
	pReq.Header.Set("Sec-Fetch-Dest", "empty")
	pReq.Header.Set("Sec-Fetch-Mode", "cors")
	pReq.Header.Set("Sec-Fetch-Site", "same-site")
	return pReq
}

There are several pitfalls here. The first and most understandable is static User-Agent. In the long term, it’s better to do it at least random. The second and less obvious point is the implementation headers in the Go language itself. The order of the headers that I specified in the code is correct, but the structure itself Header is a mapa map[string][]string in Go, which will automatically reduce all headers to a random order. If the server is monitoring this, then it will be able to identify us and ban us very quickly. One of the possible and least problematic solutions may be to use a non-standard http package.

Producer

Let's start writing adapters with IAdapterProducer, because this will be the simplest adapter in our implementation, where just one request is enough. First, let’s create a constructor into which we’ll pass the post ID.

func NewAdaptedProducer(pPostID string) adapters.IAdaptedProducer {
	return &sAdaptedProducer{
		fPostID: pPostID,
	}
}

And then, all we need is to create a request indicating the post identifier (postId) and our message (body), by sending it to the comments URL /api/comment method POST. Message pMsg.ToString() is presented as hex encoding, so there will be no conflict with double quotes in JSON format.

func (p *sAdaptedProducer) Produce(pCtx context.Context, pMsg net_message.IMessage) error {
	reqStr := fmt.Sprintf(
		`{"postId":"%s","body":"%s"}`,
		p.fPostID,
		pMsg.ToString(),
	)

	req, err := http.NewRequestWithContext(
		pCtx,
		http.MethodPost,
		"https://api.chatingar.com/api/comment",
		bytes.NewBuffer([]byte(reqStr)),
	)
	if err != nil {
		return err
	}

	httpClient := &http.Client{Timeout: 30 * time.Second}
	resp, err := httpClient.Do(chatingar.EnrichRequest(req))
	if err != nil {
		return err
	}
	defer resp.Body.Close()

	if code := resp.StatusCode; code != http.StatusCreated {
		return fmt.Errorf("got status code = %d", code)
	}

	return nil
}

Consumer

WITH IAdaptedConsumer everything is a little more complicated, because… here we will need to take care of the number of previously read messages and constantly move along new pages in order to have time to read the current traffic.

func NewAdaptedConsumer(
	pPostID string,
	pSettings net_message.ISettings,
	pKVDatabase database.IKVDatabase,
) adapters.IAdaptedConsumer {
	return &sAdaptedConsumer{
		fPostID:     pPostID,
		fSettings:   pSettings,
		fKVDatabase: pKVDatabase,
		fMessages:   make(chan net_message.IMessage, cLimitMsgs),
	}
}

The constructor also accepts a post ID, but among other things it includes a Key-Value database (to save the status of read messages) and message settings to primarily separate messages generated by the anonymous network from other messages.

func (p *sAdaptedConsumer) Consume(pCtx context.Context) (net_message.IMessage, error) {
	// Если в очереди существуют сообщения, тогда вернуть от туда
    // одно сообщение в качестве результата
    select {
	case msg := <-p.fMessages:
		return msg, nil
	default:
	}

    // Если потребитель запустился в первый раз, тогда
	if !p.fEnabled {
		// Запросить у сервиса количество комментариев
        countComments, err := p.loadCountComments(pCtx)
		if err != nil {
			return nil, err
		}

        // Вычислить количество страниц от количества комментариев
		countPages := (countComments / cPageOffet) + 1

        // Установить в БД полученное количество страниц 
        // в качестве точки начала чтения
		if err := p.setCountPagesDB(countPages); err != nil {
			return nil, err
		}
  
		p.fEnabled = true
		return nil, nil
	}

    // Получить количество страниц из БД
	currPage, err := p.getCountPagesDB()
	if err != nil {
		return nil, err
	}

    // Загрузить с сайта комментарии со страницы currPage
	return p.loadMessage(pCtx, currPage)
}

func (p *sAdaptedConsumer) loadCountComments(pCtx context.Context) (uint64, error) {
	req, err := http.NewRequestWithContext(
		pCtx,
		http.MethodGet,
		fmt.Sprintf("https://api.chatingar.com/api/post/%s", p.fPostID),
		nil,
	)
	if err != nil {
		return 0, fmt.Errorf("failed: build request")
	}

	httpClient := &http.Client{Timeout: 30 * time.Second}
	resp, err := httpClient.Do(chatingar.EnrichRequest(req))
	if err != nil {
		return 0, fmt.Errorf("failed: bad request")
	}
	defer resp.Body.Close()

	if code := resp.StatusCode; code != http.StatusOK {
		return 0, fmt.Errorf("got status code = %d", code)
	}

	var count sCountDTO
	if err := json.NewDecoder(resp.Body).Decode(&count); err != nil {
		return 0, err
	}

	result := count.Post.CommentCount
	if result < 0 {
		return 0, errors.New("got count < 0")
	}

	return uint64(result), nil
}

func (p *sAdaptedConsumer) loadMessage(pCtx context.Context, pPage uint64) (net_message.IMessage, error) {
	req, err := http.NewRequestWithContext(
		pCtx,
		http.MethodGet,
		fmt.Sprintf(
			"https://api.chatingar.com/api/comment/%s?page=%d",
			p.fPostID,
			pPage,
		),
		nil,
	)
	if err != nil {
		return nil, fmt.Errorf("failed: build request")
	}

	httpClient := &http.Client{Timeout: 30 * time.Second}
	resp, err := httpClient.Do(chatingar.EnrichRequest(req))
	if err != nil {
		return nil, fmt.Errorf("failed: bad request")
	}
	defer resp.Body.Close()

	var messagesDTO sMessagesDTO
	if err := json.NewDecoder(resp.Body).Decode(&messagesDTO); err != nil {
		return nil, err
	}

	sizeComments := len(messagesDTO.Comments)
	if sizeComments > cPageOffet {
		return nil, errors.New("has limit pages")
	}
	if sizeComments == cPageOffet {
		if err := p.incCountPagesDB(); err != nil {
			return nil, err
		}
	}

	for _, v := range messagesDTO.Comments {
		msg, err := net_message.LoadMessage(p.fSettings, v.Body)
		if err != nil {
			continue
		}
		if err := p.rememberMessage(msg); err != nil {
			continue
		}
		p.fMessages <- msg
	}

	return nil, nil
}

func (p *sAdaptedConsumer) getCountPagesDB() (uint64, error) {
	res, err := p.fKVDatabase.Get([]byte(cDBCountKey))
	if err != nil {
		if !errors.Is(err, leveldb.ErrNotFound) {
			return 0, err
		}
		res = []byte(strconv.Itoa(0))
		if err := p.fKVDatabase.Set([]byte(cDBCountKey), res); err != nil {
			return 0, err
		}
	}
	return strconv.ParseUint(string(res), 10, 64)
}

func (p *sAdaptedConsumer) setCountPagesDB(pN uint64) error {
	return p.fKVDatabase.Set(
		[]byte(cDBCountKey),
		[]byte(strconv.FormatUint(pN, 10)),
	)
}

func (p *sAdaptedConsumer) incCountPagesDB() error {
	count, err := p.getCountPagesDB()
	if err != nil {
		return err
	}
	return p.setCountPagesDB(count + 1)
}

func (p *sAdaptedConsumer) rememberMessage(pMsg net_message.IMessage) error {
	hash := hashing.NewSHA256Hasher(pMsg.GetHash()).ToBytes()
	if _, err := p.fKVDatabase.Get(hash); err == nil {
		return errors.New("hash already exist")
	}
	return p.fKVDatabase.Set(hash, []byte{})
}

That's it, we've written the finishing touches and all that remains is to connect this creation to the processing functions. This is already a standard procedure that can be viewed Here And Here.

Testing and launch

To test the functionality of our code, I wrote several docker-compose files, and also added all these adapters to the anonymous Hidden Lake network using the application HLC (composite) for messenger (HLM) and file sharing (HLF).

I set the spam level of the service to be gentle, namely: every 5 second there is a request for the number of comments every 30 seconds an encrypted message of size is sent 8KiB (taking into account HEX encoding).

To launch this whole creation, you must first download the go-peer repository. Next, go to the directory of examples with these specific adapters and run the one you need.

$ git clone --depth=1 https://github.com/number571/go-peer.git

# Запуск HLM:
$ cd go-peer/examples/anon_messenger/docker/secret_channel/chatingar
$ make

# Запуск HLF:
$ cd go-peer/examples/anon_filesharing/docker/secret_channel/chatingar
$ make

All application applications of the Hidden Lake network continued to function successfully with traffic sent through the HTTPS server.

Conclusion

Thus, we wrote Hidden Lake network adapters for a centralized service chatingar.com, which made it possible to inject anonymized traffic into the latest. This course of action made it possible:

1. Hide traffic specific to anonymous networks within HTTPS connections on real centralized services, while maintaining the same level of security and anonymity,

2. Use a third-party service as one of the ways NAT traversal for P2P communications without renting VPS or using your own resources.

All adapters we have written are completely in the public domain, the source code of which can be viewed Here. You can launch and test the functionality of the adapters via docker-compose here (HLM) And here (HLF). You can learn more about the go-peer project and the Hidden Lake anonymous network in the documentation link.