How I read all the content on RSS translated

GitHub, noticed that there are already some bots that can convert RSS from sites into messages. As an example.

• + On the plus side: texts and various media will most likely be uploaded to the Telegram cloud and remain there “forever.” Well, at least obviously longer than in a CDN;

• – This solution has disadvantages: the feed is published in a personal dialogue with a bot or in a group, it is poorly tagged and labeled, and search is difficult. The single feed is frequently updated, so you can accidentally knock off the reading area and fly down somewhere. Even if there are several groups with different sources of content, it is still difficult to systematize the materials and export them somewhere else – Telegram is also a platform interested in capturing the audience within itself. Pavel Durov also has very serious plans for a Premium subscription, which I am not going to purchase, and sometimes, for the sake of the benefits of the subscription, developers take away functionality from free clients. Who knows, what if bots become paid within 10 years? And in general, there is a risk that Telegram will disappear or (e)evolve into something that will be incompatible with my task.

RSS (and Atom).

• + Easy to create, easy to subscribe. Tim Burners-Lee is pleased with your contribution to the decentralization of the Web, you have been given a cat-web1.0-wife. All other advantages fall entirely on the client who will read the feeds. Freedom, crystal clear and identical to straight people! Just like in the 00s;

• – All modern social networks and instant messengers will resist as strongly as possible the consumption of “their” content outside their jurisdiction, because it’s hard to make money on advertising through RSS, agree? It will be a tough battle.

Selected second due to decentralization, maturity of technology and simply due to the fact that it is stupid XML, simple and understandable. Not some B2C SaaS that exists today, but tomorrow it will disappear or be protected by a paywall. Great tool, simple and reliable.

Hypothesis: RSS can combine content from all sources into any RSS feed reader. After all, a feed is just a subset of XML, and the server for spitting out XML can be coded yourself, as a last resort.

Requirements: A “single window” for reading content should be available on any popular platform (Windows (web) and Android), and should also support receiving information from 3000 sources or more, ideally there should be no restrictions on the number of RSS feeds read.

Useful resources, client selection and work flow

Where can you dive into the world of RSS?

Well. I’m familiar with RSS, there are already some subscriptions in Feedly, they are read, but somehow everything is not serious.

Even at first I didn’t know how to approach it thoroughly. Let's read about RSS by going through the list of popular resources:

• All-about-RSS: the best collection on the Internet of all services, utilities and sources that relate to this technology;

• Topic on w3bsit3-dns.com somewhere under Tekhnotrepalka: everything will be explained, prompted, for beginners it’s a rite of passage (you have to spit on the Feedly icon and publicly renounce it);

• r/RSSto read about the latest announcements and current user responses.

We read thoughtfully for a day or two, imagining a schematic diagram of the formation of feeds.
Thanks to Web 2.0, where, after the injection of hundreds of oil somewhere from the beginning of the 10s, the connectivity between large platforms has weakened so much that I need to thinkhow to extract content from some social network!
Since social networks, instant messengers and other platforms for publishing content try to keep the audience within themselves for as long as possible, they are very poorly integrated with the rest of the web, but you can programmatically automate web scraping of texts and pictures even from them.

Selecting a reader

Let's also take a look at cross-platform RSS readers. Basically, after a little research, there are only two candidates left:

1. ❌ Feedly – draconian restrictions on the free plan, the paid plan is only for a year and then has restrictions on the number of sources. There are various integrations, local AI, threat hunting, tracking brand mentions. In general, a mature, reliable and expensive solution, for which (so the authors think) you can pay more;

2. ✅ InoReader – slightly less draconian restrictions, a paid tariff with a discount is quite affordable, a wide opportunity for various automations, rules, filters, and extras. services such as tagging materials, favorites, reading lists, forwarding, re-exporting feeds, social functions, collections, and so on and so forth… And most importantly, there is no limit on the number of sources, you can also re-export the feeds you create to someone else.

Feedbin, Feeder, Digg and other cloud solutions were no longer needed due to immaturity or fewer features and limits at a comparable price.
I refused self-hosted solutions, because in this case, supporting the server, purchasing hardware, renewals and other routine things fall on my shoulders, and they already have dozens of projects that I have to support. Another minus is that, as a rule, self-hosted ones require a small amount of permanent memory to store the database. With 700 sources with their images, you need a server with at least 100 GB of memory – this is if you take it with a strong margin. And then the database will have to be cleaned regularly, once every couple of years, I'm afraid…

As you understand, InoReader was selected. Sorry, Feedly, but your main competitor is putting everyone on the back foot, given my demands for a large number of sources.
I donated to the annual tariff so that the toad would force me to use it and earn back every Eurobuck spent.

Who will parse all this with a fork?

In short, basically RSS-Bridge – a self-hosted service for conveniently “ripping” content from anywhere (if the community bothers to write to someone about it), there is public servers for different locations. All my sources of information are supported by it: VK**, FB*, IG*, X, TG… convenient! It puts anyone on their knees PolitePol, RSS.app, TT-RSS and other “rip-offs” of content for financial and technical accessibility.

RSS-Bridge can provide feeds in the form of Atom, MRSS (I use it by default), JSON, SFeed and simply in PlainText.

It was decided to do this:

1. RSS-Bridge rips out content, generates RSS;

2. InoReader subscribes to it.

Is everyone happy? Almost…

My struggle (with VK**)

The morning has begun in the village… Based on the results of two-day tests of InoReader with links to VKontakte** in the RSS-Bridge link, it was noticed that about half of the requests fail due to the aggressive protection of the social network. But these are only 150 sources out of 300. By the way, at that time I still had no idea about other public instances of RSS-Bridge on other domains and locations, somehow the information passed by.

But I met gold – this woxcab/vkrss, PHP server for issuing posts from VK** to RSS. Very detailed instructions included. Minus – not dockerized.

Forked, in two days I packaged this project into Docker as best I could, so that it can be touched and deployed Here (later salt in master). Bonus can be hidden access_token from the VK** application to .env-file.
Rented cheap server at PureServers, filled it up and launched it. Works!

How to use (briefly, otherwise read instructions):

0. Own your domain, specify the A-record on your server IP;

1. Register Standalone-application in VK**;

2. Let's pass Authorization Flow (not Implicit Flow) in two stages, we get a perpetual token;

3. We enter the token and other information in .env-file;

4. Let's launch docker compose up -d certbot webserver to obtain certificates;

5. After successfully receiving SSL certificates, uncomment sed -i 's/### //' templates/default.conf.template SSL section in the NGINX config;

6. We launch fully: docker compose restart webserver && docker compose up -d php.

Now the server will produce RSS for any request a la: https://<ВАШ_ДОМЕН>:8082/api/vkrss/index.php?id=<ЮЗЕРНЕЙМ_ПАБЛИКА_ВК>&vkrss_access_token=<ВАШ_ТОКЕН_ДОСТУПА_В_DOTENV>. Almost everything in this URL is configurable.

The execution speed is quite slow: 3-7 seconds – but not critical for InoReader. Not a single error when updating 300 feeds in VK!

Ru-Board Strikes Back

A legendary forum that, even in the age of dominance of social networks and instant messengers and after internal turmoil, continues live out live!

The problem is that the RSS feed they provide is very poorly customizable – there is no normal sorting, you cannot subscribe to messages within a specific topic. And that’s exactly what I need. Well, the engine is self-written, what do you tell it to do?

Fortunately, there was a craftsman who made most of the work is done for me. The same RSS-Bridge helped!

Reddit still supports RSSdespite further covering up third-party applications and its API, and all this for the sake of entering an IPO.

What about the rest?

There were no problems with them. RSS-Bridge the head of everything! You type Telegram in there, for example, and it pulls content from Telegram like it’s cute. The main thing is that the channel is public. There are also 0 problems with other forums – they do not have mega-threads, like in Ru-Board, and the forums can provide subscriptions to newly created topics.

Youtube subscriptions were imported with two clicks. I signed the newsletters by creating a separate mailbox for each (InoReader has such a feature) and re-subscribing for weekly newsletters there.

Helpful tips for bulk importing into InoReader (and other services that support OPML import)

Inoreader has a feature for bulk import of links to feeds, but only in the form of OPML files. If you're too lazy to manually import your subscriptions, here's what you can do…

1. During the transfer process, it turned out that I was too lazy to perform manual operations, so I wrote a couple of scripts:

   1. You can export all links from a specific VK feed**: https://gist.github.com/Kenya-West/d2fc748958a88e8b7288bba6179f2b7f;

   2. You can export all links from a specific TG folder: https://gist.github.com/Kenya-West/0e489eb750bbdd240d372032c8b7d595.

2. As soon as I received the necessary public usernames in VK**, I formed links from them:

   1. For VK**: https://<ВАШ_ДОМЕН>:8082/api/vkrss/index.php?id=<ЮЗЕРНЕЙМ_ПАБЛИКА_ВК>&vkrss_access_token=<ВАШ_ТОКЕН_ДОСТУПА_В_DOTENV>well, the same one from the section above.

   2. For TG: https://rss-bridge.org/bridge01/?action=display&bridge=TelegramBridge&username=%40<ЮЗЕРНЕЙМ_ПАБЛИКА_ТГ>&format=Mrss.

3. I inserted a list of these links, separated by a new line, into the service OPML Maker and received the finished OPML file. You can create multiple OPML files, each for a specific folder in InoReader;

4. Imported this case into InoReader. For VK** I had to wait quite a long time – for each public feed it took up to 7 seconds to form on my server, and there are 300 such feeds. But in the end, everything was successfully imported!

Improved quality of life

All the problems listed below apply to the web version, since in the mobile InoReader you can easily click on the preview, switch to another mobile application and watch the content there.

Telegram

TG turned out better than I expected! RSS-Bridge displays a feed with a built-in tag <video>, which you can watch by clicking on the picture… but nothing happens. It turned out that you need to manually enable the video controls using the right mouse button and manually start it.
Brilliant, InoReader, simply brilliant! This means we need to automate this.

VK**

With VK** it turned out more difficult. The video is definitely not present on the tape, but there is a picture with a link to the video. It would be possible to give birth <iframe> to insert a video, but everything, as always, is more complicated than you think – for embedded content you also need a parameter hashwhich can be obtained via the VKontakte API**.
We create Another Standalone application for yourself or use a ready-made one VKHost for getting access_token. If there are any bugs during authorization (for example, VK forgot to issue a grant for video), can be specified in redirect_uri=https://oauth.vk.com/blank.htmlbecause sometime in February the world's best authorization API broke and still hasn't been fixed.

The solution to these two problems is ready-made userscript

Filed it down script (need Tampermonkey or equivalent) that solves these two problems with TG and VK**:

1. Autoplays the video and enables controls for it in InoReader;

2. Instead of a picture, it substitutes a video from VK (after installing the script, you need to click on the Tampermonkey icon, select the script and click Add VK** Access Token and paste the token).

If already installed Tampermonkey: Install | Greasyfork (cuming soon…)

In the future, I plan to include embed from Telegram, since images from its CDN don’t last long…

Results

In general, you can check for yourself how much news accumulates on average per day:

Well, good luck to me trying to read all these memes, news, articles, etc.

All subscriptions are working stably, there are no errors yet. This is based on the results of a week of operation.

I dealt with the habit of checking news on TG, Reddit and VK** the very next day, forcing myself to sit in InoReader. Literally in one day I got used to its interface and, as you can see, I slightly improved its UX to suit myself.

Criticism of the decision

Well, you say: “The cloud again!”, “Why not self-hosted?” – but I will answer you that a cloud solution in my conditions will clearly be more reliable, requiring less attention to maintenance. In the case of InoReader, there is one point of failure (well, two, if Houthis add), and some TT-RSS has a couple or three more: the network may fall off, the provider may mess up, a rocket will fly into the data center – but you don’t have backups (well, let’s say they are configured, but that’s not the point). Plus, I generally don’t have enough time to properly configure the entire pipeline in the self-hosted paradigm. After all, if it took up to a week to build a cloud pipeline, how long will it take to deploy, configure and test all the solutions on your server?

There are really no problems with getting used to it, I mastered everything in one day, learned the hotkeys and then I just sit in it. It was surprisingly easy to force myself to use InoReader – for some reason I managed to trick my brain and not wait 21 days until a habit will form. In general, I “jumped over” very easily.

In fact, my solution is not a purely cloud one, but a hybrid one, since most of the content (VK** and TG subscriptions) is generated by intermediate services woxcab/vkrss And RSS-Bridge. But there is a reason for this – the original platforms are closed as much as possible from the external web and do not provide RSS feeds, so feeds are not generated directly.

conclusions

My Way of the Samurai is recommended to all those who are tired of constantly switching between different services to read content. Of course, it is much more convenient to read everything in one place. You will most likely have no problems getting used to InoReader. If there are not enough free limits, then you can pay for a month or a year. Ripping out content from VK** and TG is difficult, but possible. A similar migration using my article may take you less time than it did me (less than a week). I think that my bundle was the best choice out of the existing ones, taking into account my requirements.

Future plans

1. There is a problem with formulas and drawings in the form of transparent PNGs on a dark theme:

You need to find a JS library to define the palette in the image and then, matching the pattern, apply a CSS rule to the image filter: invert(1) and write a separate userscript for this;

2. Looking in r/RSSfound cool work flow to consume content. We'll have to study it later;

3. GitHub has learned to show the activity feed (sorry, algorithmic) on the main page, we should learn to parse it and show it, well, or wait until someone else takes care of this problem;

4. I would also like to have a separate one with a brief confidential summary of how many people wrote to me in private messages, the state of my mailbox in the morning, the weather outside, how many new messages appeared in Telegram and Whatsapp* chats and on what topics. Probably, you will need to create userbots (there are such libraries) and run a separate neuron through their data arrays so that it summarizes all these messages and somehow packages them into a beautiful report;

5. The same thing with a neural network can be implemented for other RSS feeds, for example, news ones. But this is clearly a short-lived endeavor, since I don’t have the appropriate capacity and, most importantly, time.

---

Bon appetit in consuming content!

---

* The development company, Meta, Inc., is accused of terrorism, extremism, pessimism, flatulence and other sins punishable in the legal framework of the Russian Federation
** Belongs to organized crime groups Mail.ru Group
***They're poop