dbpedia-entity - Faegheh Hasibihasibi.com/files/posters/dbpedia-entity.pdf · c DBPEDIA-ENTITY V2:...
1
c DBPEDIA-ENTITY V2: A TEST COLLECTION FOR ENTITY SEARCH • DBpedia-entity v1* has been used as a standard test collection for entity search • DBpedia-entity v2 is the updated version - Based on a recent DBpedia dump (2015-10, with 6.2M entities) - Graded relevance judgments and explicit non-relevant judgments for all queries • Consistent in collecting relevance judgments (vs. heterogeneous judgments in v1) - Results pooled from baselines, state-of-the-art retrieval methods, and old qrels - Collecting judgments using crowd workers as well as expert annotators • What’s next? Show that your model can outperform existing approaches - According to our results, recent models can hardly beat traditional baselines Chenyan Xiong Carnegie Mellon University Fedor Nikolaev Wayne State University Krisztian Balog University of Stavanger Faegheh Hasibi Norwegian University of Science and Technology Svein Erik Bratsberg Norwegian University of Science and Technology Alexander Kotov Wayne State University Jamie Callan Carnegie Mellon University Entity search DBpedia female rock singers Joan Jett Amy Lee Janis Joplin Tina Turner + DBpedia-Entity collection ➜ tiny.cc/dbpedia-entity Query Categories #queries SemSearch ES 113 named entity queries; e.g.,“brooklyn bridge” INEX-LD 99 keyword queries; e.g., “electronic music genres” ListSearch 115 list queries; e.g., “professional sports teams in Philadelphia” QALD-2 140 natural language queries; e.g.,“Who is the mayor of Berlin?” Total 467 SUMMARY QUERIES AND RESULTS Method v1 v2 MAP P@10 MAP nDCG@10 BM25 0.0884 0.0971 0.1893 0.3582 LM 0.1709 0.1837 0.3144 0.4182 SDM 0.1860 0.1880 0.3259 0.4185 LTR 0.1723 0.1831 0.2446 0.3464 BM25-CA 0.2067 0.2056 0.3265 0.4231 FSDM 0.2069 0.2039 0.3279 0.4267 BM25F-CA 0.2088 0.2126 0.3361 0.4378 FSDM+ELR 0.2210 0.2089 0.3295 0.4335 * K. Balog and R. Neumayer. 2013. “A Test Collection for Entity Search in DBpedia.” In SIGIR ’13.
Transcript of dbpedia-entity - Faegheh Hasibihasibi.com/files/posters/dbpedia-entity.pdf · c DBPEDIA-ENTITY V2:...
c
DBPEDIA-ENTITY V2: A TEST COLLECTION FOR ENTITY SEARCH
• DBpedia-entity v1* has been used as a standard test collection for entity search
• DBpedia-entity v2 is the updated version- Based on a recent DBpedia dump (2015-10, with 6.2M entities) - Graded relevance judgments and explicit non-relevant judgments for all queries
• Consistent in collecting relevance judgments (vs. heterogeneous judgments in v1)- Results pooled from baselines, state-of-the-art retrieval methods, and old qrels - Collecting judgments using crowd workers as well as expert annotators
• What’s next? Show that your model can outperform existing approaches - According to our results, recent models can hardly beat traditional baselines
Chenyan Xiong Carnegie Mellon University
Fedor Nikolaev Wayne State University
Krisztian Balog University of Stavanger
Faegheh Hasibi Norwegian University of Science and Technology
Svein Erik Bratsberg Norwegian University of Science and Technology
Alexander Kotov Wayne State University
Jamie Callan Carnegie Mellon University
Entity searchDBpediafemale rock singers
Singers Female
Joan Jett Amy Lee Janis Joplin
1943–1970
Tina Turner Pat Benatar Stevie Nicks Ann Wilson Alanis
Morissette
Lzzy Hale Debbie Harry Chrissie
Hynde
List of female rock singers Wikipediahttps://en.wikipedia.org/wiki/List_of_female_rock_singers
A. Tori Amos. Fiona Apple. Melissa Auf der Maur. Pat Benatar. Miki Berenyi. Bif Naked. Björk.Lydia Canaan.
Category:American female rock singers Wikipediahttps://en.wikipedia.org/wiki/Category:American_female_rock_singers
Pages in category "American female rock singers". The following 168 pages are in this category, out of168 total. This list may not reflect recent changes (learn ...
Women in rock Wikipediahttps://en.wikipedia.org/wiki/Women_in_rock
Jump to Singers Singers in popular music perform the vocals for bands and other ... She has statedthat "[I]t's invisible, what women do," and "[I]t's not ...
Category:Female rock singers Wikipediahttps://en.wikipedia.org/wiki/Category:Female_rock_singers
A. Kelli Ali. Zayra Alvarez. Amber (singer) Merrie Amsterburg. Anouk (singer) Joan Armatrading.Aylin Aslım. Atlanta (singer)
26 FemaleFronted Bands That Rule | Billboardwww.billboard.com/.../femalefrontedbandsnodoubtgarbagethecarpentersthepr...
Apr 29, 2015 The history of rock includes a myriad of collectives in which a strong female ... Here are26 of our favorite bands fronted by inspiring female artists. ... Gladys Knight and her crew of malebackup vocalists helped shape the ...
10 Female Rock Stars Who Shaped the History of Rock ThoughtCo
Rock
All Images Videos News Shopping More Settings Tools
female rock singers
Singers Female
Joan Jett Amy Lee Janis Joplin
1943–1970
Tina Turner Pat Benatar Stevie Nicks Ann Wilson Alanis
Morissette
Lzzy Hale Debbie Harry Chrissie
Hynde
List of female rock singers Wikipediahttps://en.wikipedia.org/wiki/List_of_female_rock_singers
A. Tori Amos. Fiona Apple. Melissa Auf der Maur. Pat Benatar. Miki Berenyi. Bif Naked. Björk.Lydia Canaan.
Category:American female rock singers Wikipediahttps://en.wikipedia.org/wiki/Category:American_female_rock_singers
Pages in category "American female rock singers". The following 168 pages are in this category, out of168 total. This list may not reflect recent changes (learn ...
Women in rock Wikipediahttps://en.wikipedia.org/wiki/Women_in_rock
Jump to Singers Singers in popular music perform the vocals for bands and other ... She has statedthat "[I]t's invisible, what women do," and "[I]t's not ...
Category:Female rock singers Wikipediahttps://en.wikipedia.org/wiki/Category:Female_rock_singers
A. Kelli Ali. Zayra Alvarez. Amber (singer) Merrie Amsterburg. Anouk (singer) Joan Armatrading.Aylin Aslım. Atlanta (singer)
26 FemaleFronted Bands That Rule | Billboardwww.billboard.com/.../femalefrontedbandsnodoubtgarbagethecarpentersthepr...
Apr 29, 2015 The history of rock includes a myriad of collectives in which a strong female ... Here are26 of our favorite bands fronted by inspiring female artists. ... Gladys Knight and her crew of malebackup vocalists helped shape the ...
10 Female Rock Stars Who Shaped the History of Rock ThoughtCo
Rock
All Images Videos News Shopping More Settings Tools
female rock singers
Singers Female
Joan Jett Amy Lee Janis Joplin
1943–1970
Tina Turner Pat Benatar Stevie Nicks Ann Wilson Alanis
Morissette
Lzzy Hale Debbie Harry Chrissie
Hynde
List of female rock singers Wikipediahttps://en.wikipedia.org/wiki/List_of_female_rock_singers
A. Tori Amos. Fiona Apple. Melissa Auf der Maur. Pat Benatar. Miki Berenyi. Bif Naked. Björk.Lydia Canaan.
Category:American female rock singers Wikipediahttps://en.wikipedia.org/wiki/Category:American_female_rock_singers
Pages in category "American female rock singers". The following 168 pages are in this category, out of168 total. This list may not reflect recent changes (learn ...
Women in rock Wikipediahttps://en.wikipedia.org/wiki/Women_in_rock
Jump to Singers Singers in popular music perform the vocals for bands and other ... She has statedthat "[I]t's invisible, what women do," and "[I]t's not ...
Category:Female rock singers Wikipediahttps://en.wikipedia.org/wiki/Category:Female_rock_singers
A. Kelli Ali. Zayra Alvarez. Amber (singer) Merrie Amsterburg. Anouk (singer) Joan Armatrading.Aylin Aslım. Atlanta (singer)
26 FemaleFronted Bands That Rule | Billboardwww.billboard.com/.../femalefrontedbandsnodoubtgarbagethecarpentersthepr...
Apr 29, 2015 The history of rock includes a myriad of collectives in which a strong female ... Here are26 of our favorite bands fronted by inspiring female artists. ... Gladys Knight and her crew of malebackup vocalists helped shape the ...
10 Female Rock Stars Who Shaped the History of Rock ThoughtCo
Rock
All Images Videos News Shopping More Settings Tools
female rock singers
Singers Female
Joan Jett Amy Lee Janis Joplin
1943–1970
Tina Turner Pat Benatar Stevie Nicks Ann Wilson Alanis
Morissette
Lzzy Hale Debbie Harry Chrissie
Hynde
List of female rock singers Wikipediahttps://en.wikipedia.org/wiki/List_of_female_rock_singers
A. Tori Amos. Fiona Apple. Melissa Auf der Maur. Pat Benatar. Miki Berenyi. Bif Naked. Björk.Lydia Canaan.
Category:American female rock singers Wikipediahttps://en.wikipedia.org/wiki/Category:American_female_rock_singers
Pages in category "American female rock singers". The following 168 pages are in this category, out of168 total. This list may not reflect recent changes (learn ...
Women in rock Wikipediahttps://en.wikipedia.org/wiki/Women_in_rock
Jump to Singers Singers in popular music perform the vocals for bands and other ... She has statedthat "[I]t's invisible, what women do," and "[I]t's not ...
Category:Female rock singers Wikipediahttps://en.wikipedia.org/wiki/Category:Female_rock_singers
A. Kelli Ali. Zayra Alvarez. Amber (singer) Merrie Amsterburg. Anouk (singer) Joan Armatrading.Aylin Aslım. Atlanta (singer)
26 FemaleFronted Bands That Rule | Billboardwww.billboard.com/.../femalefrontedbandsnodoubtgarbagethecarpentersthepr...
Apr 29, 2015 The history of rock includes a myriad of collectives in which a strong female ... Here are26 of our favorite bands fronted by inspiring female artists. ... Gladys Knight and her crew of malebackup vocalists helped shape the ...
10 Female Rock Stars Who Shaped the History of Rock ThoughtCo
Rock
All Images Videos News Shopping More Settings Tools
female rock singers
Joan Jett
Amy Lee
Janis Joplin
Tina Turner
+ DBpedia-Entity collection
➜
tiny.cc/dbpedia-entity
Query Categories #queriesSemSearch ES 113named entity queries; e.g.,“brooklyn bridge”
INEX-LD 99keyword queries; e.g., “electronic music genres”
ListSearch 115list queries; e.g., “professional sports teams in Philadelphia”
QALD-2 140natural language queries; e.g.,“Who is the mayor of Berlin?”
Total 467
SUMMARY
QUERIES AND RESULTS
Method v1 v2MAP P@10 MAP nDCG@10
BM25 0.0884 0.0971 0.1893 0.3582LM 0.1709 0.1837 0.3144 0.4182SDM 0.1860 0.1880 0.3259 0.4185LTR 0.1723 0.1831 0.2446 0.3464
BM25-CA 0.2067 0.2056 0.3265 0.4231
FSDM 0.2069 0.2039 0.3279 0.4267BM25F-CA 0.2088 0.2126 0.3361 0.4378FSDM+ELR 0.2210 0.2089 0.3295 0.4335
* K. Balog and R. Neumayer. 2013. “A Test Collection for Entity Search in DBpedia.” In SIGIR ’13.
