Exploring family relations from online obituaries using text mining and data visualisation tools

Using a copy of all the obituaries published online by the Ahram newspaper from January 2002 till April 2008 it is possible to use Linux command line tools (gawk, sed, bash) to find family relations between individuals in certain professions. An example given here explores the family links between a sample of 456 Egyptian state security officers.

This is a very brief description of the method.

  1. The first step is to convert the HTML files downloaded by curl into one giant text file.
  2. Then to move each separate obituary into a line of it's own.
  3. Extract officer names sandwiched between rank and place of work into a separate text file.
  4. Search for the names of each officer through each obituary, family links between different officers can be discovered.
  5. The output is in GraphViz .dot format, which draws a graph similar to the one below.
  6. Graph showing 63 family links between 174 officers from 456 Egyptian state security officers. Each link corresponds to a family tie of a variable degree of relationship including in-laws.

    Graph showing 63 family links between 174 officers from 456 Egyptian state security officers. Each link corresponds to a family tie of a variable degree of relationship including in-laws.

    This is just a preview of what might be possible using a data set of 43,156 obituary. Without control group(s), this graph says nothing other than it's pretty and that there are family links between officers. Other methods for analysis of data could be done using statistical methods to answer different questions.

    UPDATE: I attached the list of SS officers and the script used to find links between officers and output a .dot Graphviz file. You can download the obituaries dataset from here.

The NDP Synagogue 2

Lina Attalah and Mohamed El Dahshan of Al-Masry Al-Youm followed up on my post about the Magen Abraham synagogue that was turned into an NDP office in Hadayeq el-Quba. They did a very good job in finding answers to many of the questions regarding why this building is in such a horrible state.

the interior of hte synagogue

“This temple was built by the Adda family,” says Carmen Weinstein, president of the Egyptian Jewish Community Council (JCC).

"The Adda's were a Jewish Egyptian family of industrialists and bankers, who contributed to the growth of the Egyptian economy in the 1940s. I wish the state would preserve this temple, which is unique in this part of the city." For Jewish communities in Egypt before 1952 it was customary to erect neighborhood synagogues. "In each neighborhood, the local Jewish collectivity built a synagogue," says Weinstein, who also points out that while 29 synagogues once existed in Cairo, only 13 remain.

When contacted for more information about the building, Cairo Governorate officials expressed surprise with respect to the building's unusual status, and gratitude that Al-Masry Al-Youm reporters brought the matter to their attention. According to one governorate employee who wanted to remain anonymous, the building is recognized in the district's files, but has no licenses or ownership documents.

"This means that no measures of demolition, or restoration, have been taken with respect to the building. Nevertheless, the Hadayeq el-Qubba district headquarters believe the construction to be stable and safe," he says. "Since there has been no ownership documents for the synagogue, the government has put its hand on it."

Madness constitutes only 15% of all psychiatric diagnoses

Severe mental disorders constitute only 15% of all the psychiatric diagnoses in the ICD-10

The hypothetical column on the left is meant to show that as a result of the stigma of mental illness, most people think that mental illnesses are always severe.

Delirium is a totally reversible state, people with dementia need long term care and can never be more harmless. Lots of people recover from psychosis totally and some retain symptoms.

بلال فضل و علاج التوحد بالدايت الأونطة

كتب بلال فضل في المصري اليوم يوم 12/10/2009 عن طبيبة سورية تعالج التوحد بعلاج غذائي مُعين و ينتقد وزارة الصحة المصرية في عدم توفير هذا العلاج للمرضى المصريين و عدم توفير تحليل للكشف المبكر للمرض.

الطبيبة أسمها هيفاء الكيالى و مكتوب على موقعها أنها حصلت على ماجستير الكيمياء الحيوية و خبيرة بالعلاج بلأعشاب و عضوة جمعية المخترعين السوريين و " لها عدة لقاءات في الصحف والمجلات ولها العديد من المقابلات التلفزيونية".

التوحد من الأمراض النفسية الشديدة التي تصيب الأطفال و تعوق نمو الأعصاب الدماغية الخاصة بالتواصل مع الأخرين. و يشخص في السنين الأولى من العمر. و الحقيقة أنه تعددت النظريات حول المرض و أتهمت أشياء كثيرة بأنها تسبب للمرض مثل العوامل البيئية و الغذاء و المواد التي يتعرض لهل الحوامل في الأسابيع الأولى من الحمل. و الأرجح أن المرض سببة طفرة جينية في عدد كبير من الجينات (مثل أغلب الأمراض النفسية).

في الولايات المتحدة والمملكة المتحدة و بعض الدول الأخرى قامت حملة شرسة من أولياء الأمور و أطباء الطب البديل و الإعلام يإتهام تطعيم الحصبة (الثلاثي) بأنة السبب. و لا أعلم كيف أنتشرت تلك المعلومة و لكنها كانت قوية لدرجة أن تطعيم أحد أطفال توني بلير كان خبر هام جدا و للأسف زادت أعداد الأصابة بالحصبة. و نشرت دراسة ممتازة في الدورية المفتوحة بلوس وان تؤكد أن تطعيم الحصبة لا يزيد من فرص حدوث التوحد.

التوحد ليس له علاج و العلاجات السلوكية و الدوائية كلها للأعراض الثانوية للمرض و تحسين جودة الحياة.

ولا يوجد تحليل للمرض حتى الآن و كلها مازلت أبحاث أولية جدا تحوال إيجاد علاقة بين مواد في الجسم مختلفة و إن كانت تعطي مؤشر عن وجود المرض أم لا.

و للأسف تظهر العلاجات البديلة لتعطي أمل كاذب لأولياء الأمور. فظهر علاج يمنع الأطفال من تناول الجلوتن (بروتين في القمح و الشعير) و الكازيين(بروتين الألبان).

The NDP synagogue

In April 2008, in Hadayeq el Qobba I stumbled upon this synagogue which was turned into a public affairs office, NDP office, a nursery and a small mosque.

Taking photos is strictly prohibited. But I managed to snap some shots while claiming that I am studying architecture and interested in the building style.

This is apparently the most looked after synagogue in Egypt. Thanks to Farouk Hosni. Probably the next president of the UNESCO.

UPDATE: Yes, it is "the most looked after" synagogue as in the wooden floors are in a horrible shape, cables coming and going everywhere, horrible neon lights, overstuffed filing cabinets piled over each other rusting, broken window panes replaced with cardboard and the whole sanctity of the place, that was once there and can be vaguely felt radiating from the large dome and the star of David windows, is replaced with a grim dark grey Mogama3 feel.

UPDATE 2: Lina Atallah and Mohamed El Dahshan of Al-Masry English went and investigated this synagogue: The curious case of a Cairo synagogue.

معبد الحزب الوطني


Blindfolding is commonly used by security forces that physically torture people. It works by making the victim disoriented and fearful. A person will lose his orientation to place and the sense of direction first. Then, if applied long enough for more than a day, it can make some experience visual hallucinations. It makes a blow, a kick or an electric shock unpredictable and thus far more distressful.

These were some of the effects on the victim and they were quite straightforward. But what about the effects on the perpetrator?

Here is what a former Israeli military commander said in an interview to the BBC:

''Maybe [the kid is] blindfolded for him not to see the base and how we're working... But I believe maybe we put the blindfold because we don't want to see his eyes. You don't want him to look at us - you know, beg us to stop, or cry in front of us. It's a lot easier if we don't see his eyes.

''When the kid is sitting there in the base, I didn't do it, but nobody is thinking of him as a kid, you know - if there is someone blindfolded and handcuffed, he's probably done something really bad. It's OK to slap him, it's OK to spit on him, it's OK to kick him sometimes. It doesn't really matter.''