As we did alert about damage to subway



A long time ago, about a year ago, when the Moscow metro was broken in random places, and surprisingly often, we ( dcoder_mm & Irenica ) the idea to make some sort of a service to alert you to breakdowns.

This idea may seem strange, dcoder_mm too so it seemed, until then while itself one of these breakdowns were not included. Standing on a crowded platform for 10 minutes waiting for a train, as it turned out unpleasant, so much so that in the future decided not to get caught.

After this incident, decided to look through twitter: I write anything about the incident in the subway. As it turned out, writing. The first tweets about this was about 10 minutes before we went on the subway.

Then slowly and began to emerge an idea: collect information from twitter, looking for mention of the breakdowns of the metro and warn the user if something is found.

But then this idea was abandoned for a long time, but in the fall of 2014, we again thought of her metro again started to break down.

As the notification method, it was decided to use SMS. You could of course use something else, though Twitter bot to aggregate messages on metro start, but SMS has one advantage — they reach even when there is no normal Internet connection. And in the subway, it is often not the case, but GSM is tuned normally. (When we started to do service, Wi-Fi throughout the subway was not).

The case remained for small — to do parilku tweets and SMS.

To check the battle was written a simple one-line script that gets through the search (curl search.twitter.com) the latest tweets with the hashtag #metro, highlighting the one you want by key words, and sending us a SMS.

the 
curl -ssl "https://twitter.com/search?f=realtime&q=%23%D0%BC%D0%B5%D1%82%D1%80%D0%BE&src=typd" | grep -E -o "js-tweet-text tweet-text.*<\/p>" | sed -e 's;[><\"=-]; ;g;s;js tweet-text tweet-text lang EN data aria-label-part 0;;g;s;/a;;g;s;\/\(a\|p\); ;g;s;\(.span\|.strong\|class\|twitter\|timeline\|link\|js display-url\|invisible|\tco ellipsis\|href\|nofollow\|dir\|ltr\|data\|expanded\|url\|invisible\|tco\|ellipsis\|google&amp\;utm_medium\|banner&amp\;utm_campaign\|business_news\|target\|_blank\|title\|atreply\|pretty\|&nbsp\;\|rel\|s\|?utm_source\|draggable\|false\|alt\|aria\|label\|u\|hidden\|pre\|embedded|\true\|b\|a\|qery\|orce\|hahtag_click\|hahtag\|j\|nav\|emedded\|tre\|&qot\;\|emedded\|tre\|rc\|hh\|qery\|orce\|hhtg_click\|hhtg\|img\);;g;s; ; ;g;s;
\; ;;g;s;&qot\;;;g;s;emedded;;g;s;tre;;g;s; \/ ;;g;s;hhtg_click hhtg;;g;s;hh;;g;s;qery oe;;g;s;\/tg\/[a-zA-Z]*;;g;s;tweet;;g;s;text;;g;s;lng;;g;s;nd;;g;s;prt;;g;s;http://intgrm.com\s\/[a-zA-Z]*\/;;g;s;[A-Z%a-z]*;;g;s;[\/_\/:\/?\.@&;...]*;;g;s;^[0-9]*;;g;s;[0-9]\{3,\};;g;s;^\s*;;g;s;*; a ;g;s;^[0-9] ;;g;s;# ;#;g;s;[0-9 #]*$;;g' | grep -E-i "victims|accident|emergency|problem|will rabotat|SECR|Perabo|astral|interval|katastrofa|perekryt|ustanovlen|prosteel|BITKOM|Satik|interval|don't go|repair|suspended|failed| > > |train broke down|bombs|explosion|terrorist attack|difficult|pause|mourning|gawk|smoke|vserv|arrows|flooded|flood|flood|flooded|on the tracks|don't go|don't go|voltage" | grep -E-i "Marlin|Dostoevski|Truben|Sretensk|Chkalovsk|risk|Kristians|Dubrovka|Kozuchowski|Pechatnik|Volzhsk|Lublin|Bratislavsk|Marlin|Borisov|Chiprovski|Tablecovers|Kahovs|Lesoparku|Starokachalovskaya|Skobelevs|Ushakov|Gorchakov|Unincreased|Cernisovs|Preobrazhensk|Sokolin|Krasnoselsk|Crasn|City|Lubang|Kotn|Bibliotek|Kropotkinsk|Frunzenski|Sportive|Vorobiev|University|Vernadsky|Yugozapaden|Zapadni|South Zapadn|Tropari|Alma|Krasnogvardeisk|Domodedovsky|Orekhov|Tsaritsyn|Kantemirovs|Kashirsk|Kolomensk|Avtozavodsk|Novokuznetsk|Tetralin|Twerk|Makowski|Dynam|Aeroport|Sokol|Wojkowski|Wadn|Recn|Patrick|Main|Volokolamsk|Makinen|Strogin|Krylatski|Maladin|Konczewski|Slavyansk|Pubed|Smolensk|Arbat|Rivals|Baumans|Electrozavodsk|Semenovsk|Partizansk|Izmailovsk|Pervomaisk|Shelkovsk|Alexandrovsk|Smolensk|Vystavochno|mejdunaroden|Studentesc|Kutuzovsk|fil|bagrationowsk|Filesc|Pionersk|Beloruss|Novoslobodsky|Avenue Mir|Komsomolsk|Kursk|Thansk|Pavelec|Dobryninsky|Oktyabrsk|Kultur|Kievs|Krasnopresnenskaia|Babushkins|Sviblov|Botanicus|VDNH|Alekseevsk|Rigs|Sharesc|Turgenevs|Tretjakovs|Chabalowski|Leninsk|Academicas|Profsoyuz|Cheremushki|Kaluzhsk|Belyaev|Konkow| Mill|Yasenev|Novoyasenevski|Galabin|Lermontovs|Wihin|Razans|Kuzminki|Tekstilshik|Volgogradsk|Proletarsk|city-Gorod|Kuzneck|Pushkins|Barricade|1905|Behav|Polezhaevsky|Oktyabrsk[a-I ]pole|S[uu]cinsk|Spartak|Turinsk|Shadrinsk|Planer|Novocain|Novogireevo|Perov|Enthusiast|Aviamotors|Ilyich|Marxists|Tretjakovs|Delow|Altufiev|Bibirevo|Otradi|Vladykin|Petrovsk|Razumovsky|Timiryazevsk|Dmitrovsk|Savelovsk|Mendeleevsk|Cvet|Chehovski|Borovick|Polanc|Serpukhovsk|Tulsk|Nagatinsk|Nagorn|Nahimovski|Sevastopolis|Chertanovsky|Wina|Prisk|Angel|Annin|Dansk|Arbatsko-Pokrovsk| Barclays Premier League | blue[aiue]|Taganskaya-krasnopresnensk| TKL |purple[aou][IWA]|zamoskvoretsk| PLN |zel[it]h[aou][IWA]|high-timization| STL | ser[whoa][ail]|sokolnicheski| CL | red[aou][EIU]|Phil[her]FAC| FL |blue[aou][IWA]|ring|Kaluga-rigs| KRL | red[EUA][IWA]|Kalininsk| C |W[it]LT[aou][IWA]|orange[OAU][IWA]|lublinsky|salads| LK |kahovs|butowski" | grep -i-v -E "years ago|years ago|years nazad|vspomnil|Odess|Ukraine|SPb|Piter|Peterburg|Sankt-Peterburg|power|residen|Pamyat|Goji|aged|taxi|okna|SIM|Deng|iphone|huge|acci|skids|porn|porn|pron|pron|diet|follow|retweet|skachat|besplatn|Smoking mixture|spice|Minsk|istor|life is beautiful|kreschatnike|apartments|survived|prosecutors|Kiev |Kiev|criminal|Lesn|Gero Dnepr|Syrec|Chernigovsk|Minsk|dorohozhychi|Darnit|Obolon|Lukianovsk|Levoberezhny|Petrovic|Gold gate|hydro|Taras Shevchenko|Palace Sports|Dnepr|Contracts area|Klausk|Arsenal|Postal square|Pechersk|Khreshchatyk|Maydan Independence|Friendships People|square lion Tolstov|Vydubich|Olympic|Slavutich|Vokzal|Palace Ukraine|Osokor|Politekhnicheski Institut|Libdsk|poznak|Sulewski|Imeesc|Kharkovsk|Beresteisk|Goloseevsk|Warlic|NIWC|Vasilkovsk|Borispolska|Svyatoshin|Red farm|Zhitomirski|race track|Akademgorodok|Teremki|Paris|hospitals|Boston|voice|blood" | sort | uniq


 
And oddly enough, it was quite normal to work. Sometimes, of course, there were false positives, but most importantly, the alerts came at all more or less important damage. Came quickly, well before this information appeared in the media or anywhere else. 
 

 
Now, it is time to filter false positives. The most blatant and obvious spam, for example, the hashtag #metro did not pass, because it is not contained key words (like "accident" or "failure"). But it didn't save, for example, from reports of breakdowns of the metro in other cities (not always in a tweet on the failure of the metro in St. Petersburg, write the word "Peter"). So I had to enter the list of "stop words" that message is not sent. It  included  the names of some stations in other cities, the key words that were often found in advertising etc. 
 

 
The problem with that is that one event can have a couple dozen tweets, and much the same text, decided in the forehead: just block all the posts about this branch of the subway at a certain time. 
 

 
However, there were other causes of false positives: reports about damage to other vehicles (but with the tag "metro"), message, old damage to underground (a couple of accounts writes tweets about the tragedy in the subway in the summer of 2014 as if it just happened).
 

 
When it finally became more or less stable to work, we made a web interface with the possibility of registering by phone number. Slowly began to catch up with more people, and of course, in full accordance with Murphy's law, it happened a few false positives. So this is not repeated, made moderated: That is, first message come to us, and if we are for a few moments not forbid their sending, is sent to all other users. 
 

 
And we have statistics. Well, if you collect data, it is a sin not to have statistics. In our case, this is just a display of the number of failures on the branches for a certain period of time. And a brief description of each failure. In the future, probably will add something, first you need to collect more data.
 

 
In General, this is interesting: msk-metro.ru Here we are. We can assume that the project is still at the stage of beta testing, so if you accidentally left comes free — do not worry. We  fix  it, and next time the same SMS will not come.
 

 
And lives it all on the Raspberry Pi on debian.
Article based on information from habrahabr.ru

Comments

Post a Comment

Popular posts from this blog

Powershell and Cyrillic in the console (updated)

Image
In the development process is very often necessary to run a powershell script from a console application. What could be easier? the #test.ps1 & $PSScriptRoot\ConsoleApp.exe Examine the behavior of console applications when running them from the command line, using PowerShell and using PowerShell ISE: execution Result In PowerShell ISE, there is a problem with encoding, as the ISE expects the output to the encoding 1251. Use Google and find two problem solving: c use [Console]::OutputEncoding and through the powershell pipeline. Let's use the first solution: test2.ps1 $ErrorActionPreference = "Stop" function RunConsole($scriptBlock) { $encoding = [Console]::OutputEncoding [Console]::OutputEncoding = [System.Text.Encoding]::GetEncoding("cp866") try { &$scriptBlock } finally { [Console]::OutputEncoding = $encoding } } RunConsole { & $PSScriptRoot\ConsoleApp1.exe } execution Result At the command prompt, all is well, bu
Read more

Active/Passive PostgreSQL Cluster, using Pacemaker, Corosync

Image
Description This article describes an example of configuring Active/Passive PostgreSQL cluster, using Pacemaker, Corosync. As the disk subsystem is considered the drive from the system data storage (CSV). Similar to the Windows Failover Cluster from Microsoft. Technical details: OS Version — CentOS 7.1 Version package pacemaker — 1.1.13-10 Package version pcs — 0.9.143 PostgreSQL — 9.4.6 As servers(2pcs) — iron server 2*12 CPU/ memory 94GB As a CSV(Cluster Shared Volume) — an array of class Mid-Range Hitachi RAID 1+0 Preparation of cluster nodes Rule /etc/hosts on both hosts and make the visibility of hosts to each other by short names, for example: the [root@node1 ~]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 10.1.66.23 node1.local.lan node1 10.1.66.24 node2.local.lan node2 Also make exchange between servers using SSH keys and
Read more

Automatic deployment ElasticBeanstalk using Bitbucket Pipelines

Image
bitbucket is a service ( Bitbucket Pipelines), which among other things allows extremely simple to automate the deployment of applications in the Amazon cloud, in particular using ElasticBeanstallk. To whom interesting, I ask under kat. the Prerequirements You should already be created ElasticBeanstalk application and deployed environments (in order to do this, refer to the official AWS documentation for, so, for example, unfolds Django ). Since the service is still undergoing testing, you need to request access to it here . the project setup Let's say we have two branches, master and we want to deplot on production environment and development — development environment. Create in the root of the project file bitbucket-pipelines.yml with the following content: the image: python:2.7.11 pipelines: branches: master: - step: script: - apt-get update # to install required zip - apt-get install -y zip # required for packaging up the application - pip install b
Read more