I’d like to index emojis.
I first tried terms extraction but no option appears to suit extracting just characters.
So i’ve tried corpus terms indexer: i’ve created a list of emojis, which i’ve put in a column, then copied two times so that it’s a three-column tsv file, this way:
stem (tab) main form (tab) forms 🤡 (tab) 🤡 (tab) 🤡 😂 (tab) 😂 (tab) 😂 🤔 (tab) 🤔 (tab) 🤔
[bloody hell, copying this table here was a pain, uploading a screenshot would have helped]
I’ve run corpus terms indexer with the followin options.
- word boundaries: non-separated
- normalize lexical items: no
But after a few seconds run, i got “0 occurrences indexed”.
Is that idea at all doable?
It is true that with corpus terms extractor / indexer, it is not supposed to work. To double check, please invite me into you projet: lionel dot villard at esiee dot fr
For now, the only option I see is to use a dictionary of emoji to transform, in your initial corpus, the utf8 emoji codes into the full textual description.
We are investigating also the feasibility to extract emoji, emoji codes and descriptions, using our Name Entity Recognizer script.
I hope it helps
See indexer script at 2022-04-20 15:04:50
What i don’t understand is why the option “word boundaries: non-separated” doesn’t fit.
From what I see the original csv file with tweets (zipped in all-dates-retravaillees-tire.zip) has not been stored in UTF8 (or has been accidentally converted before zipping it). It is ANSI charset encoding: emojji have been converted mainly to “??”. So, while working with it in CorTexT Manager, there is no way to search for UTF8 emojji codes.
Finally it’s ok, you were right Lionel, i was using a wrong version of the file, whose encoding had been broken (by Excel of course, what else?)
Here are the winners: 😂🤣🇫🇷😉👍🤔
By number of distinct documents (mainly tweets during French election campaign):
A new feature has been published: https://docs.cortext.net/named-entity-recognizer/#emojis
NER script of CorTexT Manager is now able to annotate, extract and use emojis in analyses (in a network, to tag…). Among other interesting new features!