Performance testing results

Performance results for Category 18.
Size of the dataset being classified/time taken to classifiy
5000 - 0.29 seconds
10,000 - 0.60 seconds
15,000 - 1.04 seconds
20,000 - 1.17 seconds
50,000 - 2.77 seconds

Model was built using the standard parameters

Augmented Categorization results

5 Precision/recall on test set: 25.00%/32.00%
6 Precision/recall on test set: 33.33%/15.00%
14 Precision/recall on test set: 6.67%/1.33%
16 Precision/recall on test set: 11.11%/20.00%
17 Precision/recall on test set: 12.50%/5.88%
29 Precision/recall on test set: 20.00%/8.00%
37 Precision/recall on test set: 18.18%/15.38%
48 Precision/recall on test set: 0.00%/0.00%
49 Precision/recall on test set: 15.38%/20.00%
58 Precision/recall on test set: 0.00%/0.00%
60 Precision/recall on test set: 9.09%/50.00%
61 Precision/recall on test set: 11.11%/9.09%
64 Precision/recall on test set: 36.36%/10.81%

Manual categorization analysis

For several categories, it was not easy to pick samples since the categories were vague and the process would be too tedious.
For others, it was easier to use key words from the category or the samples of the category to find additional samples.
A better approach would have been to use a much bigger sample.
The expected boost from augmentation will be very slight.

Category 64 - Manually categorized queries

688716 shooting in new bern; nc
688744 shootings in little rock arkansas
492533 model" "hunter boats
787367 willie marcoft hunting guide
400622 link:http://moosehuntinginalaska.com
594541 plentyyoffish.com
387960 laughlin; nv fishing report
299517 fishing report college station
299466 fishing hooks manufacturer

Manual entries for category 61

609302 pro score gymnastics results
359732 in need of a scoreboard in texas

Manual entries for category 60

702204 spain hockey teams

Manual entries for category 58

536537 northlake college baseball 2003 roster
520295 negroleaguebaseballclowns
474932 memphis technical "high school" baseball history
66251 barry bonds baseball firsts

Manual entries for category 49

85537 black love poetry.com
772410 web site to meet married people for dating
47649 asian and black dating reno; nv
467358 maxidate internet dating service
398668 link:dating-101.org
714763 supremefriend.com

Manual entries for category 48

199785 david hale" smith
270698 famous people in dubois
590106 pinoy actors pics
725474 the bronx actor
199167 daughter's of famaus actors

Category 37 - Manually categorized queries

465117 master cuts lawn and tree service el paso; tx
681589 sensation "lawn mowers
299193 fish lawnmower repair
639235 reptiles-gardens
741091 treasure coast gardens
636041 relief society garden
323535 garden peach tomatoes" stores
317753 frommers london covent garden

Category 29 - Manually categorized queries

85537 black love poetry.com
772410 web site to meet married people for dating
47649 asian and black dating reno; nv
467358 maxidate internet dating service
398668 link:dating-101.org
714763 supremefriend.com
317425 friendsmovie
317330 friends and lovers with leon

Category 17 - Manually categorized queries

442898 lowrider car shows 2005
627253 razzle dazzle show
725867 the cosby show - 1980
474983 memrobilia show
266935 eyes on design car show
231040 dr. phil 2.2.05 show
453439 making a living tv show
505420 mtv.con switch foot
52260 atv'sonebay
56266 av.cevat zel star tv.
745255 tv chix
745460 tvcc;athens
765097 volkswagens on tv
785117 why americans love reality tv
786704 will and grace tv show
88224 blue bird tv
434339 live pause tv
412387 link:www.trtint.tv
263185 everything msntv
217613 direct tv worth $1;000.00
104076 btv 5.4.1 warez
565426 panne and televiseur and samsung and cf and 5937bn
538624 nrg television

Category 16 - Manually categorized queries

84521 bix radio 1320
94524 bosewave radio
623126 radiorama biography
123760 caribbean radio station ny 93.5
156657 classic music radio fort lauderdale florida
161907 cobra c.b. radio
164654 college radio 1980s
346545 hindman radio
379054 kkjg radio
381616 ktu internet radio
409897 link:www.moeradio.net
548716 oldies radio in toronto
622672 radio cooperation
622756 radio monitoring receiver antenna

Category 14 - Manually categorized queries

605428 preschool christian lessons and crafts
455639 mangyan crafts
44773 art & craft hobby store in mumbai
382742 la county crafts stores
240538 easter bunny milk jug craft
184415 crafting wooden arrows
453416 making "small ornaments" wood
727617 the maker of a piece of jewelry
453103 make kitty crafts
651401 roler coasters at six flags
616508 puppets cut out
537924 notseen ceramics
797001 ww.edome.net/rollercoaster

Category 6 - Manually categorized queries

492423 model golf screensaver
325609 geico lizard screensaver http
241167 easy screensaver warez
770049 water and screen and saver
40152 aquarium fish tank screen savers
469698 mcleodcomputers.com
97615 brandon computer consulting
362340 international computer electronics
26646 amd computer reviews
171100 computer courtyard
171477 computer word dictionary cell address
171575 computers and binary numbers

Category 5 - manually classified queries

The following were manually classified to be positive examples of this category:

50019 at&twiless.com
152473 cingular cellular prefixes; king county; wa
207173 dell inspiron 8100 microphone toolbar missing
453158 make phone calls on the computer
489434 missoula montana phone book
491978 mobile phonetools pc220 acheter v3 motorola
494590 monitor phone and email
499723 motorola m3788 unlocked gsm cellular mobile phone to use with sim card
564361 palmertonteephone.com
571814 payphone malaysia
582354 phone number company wallkill ny
68196 battery packs for cordless phones
701355 south western bell freedom phones
716374 swisscome mobile phone directory
453005 make a phone call on msn messenger
439891 los altos;ca phonebook
431233 links:cell-phone-info.com
387567 latest cellphone equipments
352076 how to identify withheld numbers on cell phone
314141 free ny phone numbers
89261 bluetooth packages for cellular phones
132722 cell phone number finders
115258 call phone service
106406 bullhead phone book 86442
582516 phones for sprint existing customers
721687 telecom red uk
660329 safari telecommunications
741629 tri or service or network
629913 realsimplenetwork
615122 public networks
352101 how to install wirelsss network
222129 dluxnetwork.com
56538 avaya two line telephones

Augmentation sample dataset

I took a random sample of unlabeled queries - 5000 queries in all.
These were classified using the SVM classifier.
The results were manually classified and added to the augmentation database.

Labeled instances

Following are the labeled instances of the categories that need to be augmented. These will be used to guide the manual labeling process.

Found: 4 of category=5
Computers\Networks & Telecommunication
800020 build and secure a wlan
800052 free text messages
800087 networksolutions
800116 computers networks telecommunication

Found: 4 of category=6
Computers\Other
800025 cell phone screensavers
800031 computer furniture
800071 lazarus
800119 computers other

Found: 5 of category=14
Entertainment\Other
800026 ceramics
800072 learn to make puppets
800099 rollercoasters
800107 topiary
800128 entertainment other

Found: 3 of category=16
Entertainment\Radio
800067 jackfm
800089 npr
800126 entertainment radio

Found: 4 of category=17
Entertainment\TV
800004 alfred hitchcock
800019 breakfasttelevision
800033 curb your enthusiasm
800127 entertainment tv television

Found: 5 of category=29
Living\Dating & Relationships
800054 friendships poem
800077 match.com
800079 mensrings
800092 online dating
800140 living dating relationships

Found: 4 of category=37
Living\Landscaping & Gardening
800017 bench grinders
800068 john deere gator
800107 topiary
800148 living landscaping gardening

Found: 2 of category=38
Living\Other
800020 build and secure a wlan
800154 living other

Found: 2 of category=47
Online Community\Other
800049 flash flash revolution
800160 online community other

Found: 5 of category=48
Online Community\People Search
800002 actress hildegarde
800032 crispin glover
800044 emmitt smith
800096 rafael nadal
800158 online community people search

Found: 3 of category=49
Online Community\Personal Services
800077 match.com
800092 online dating
800159 online community personal services

Found: 4 of category=57
Sports\Auto Racing
800001 1967 shelby mustang
800065 indycar
800084 nascar
800168 sports auto racing

Found: 3 of category=58
Sports\Baseball
800100 safeco field
800102 seattle mariners
800169 sports baseball

Found: 4 of category=60
Sports\Hockey
800073 lightning tickets
800075 london knights hockey club
800088 nhl allstar
800171 sports hockey

Found: 4 of category=61
Sports\News & Scores
800085 nba playoff
800086 ncaa fb scores
800088 nhl allstar
800172 sports news scores

Found: 3 of category=62
Sports\Olympic Games
800016 beijing 2008
800091 olympic history
800176 sports olympic games

Found: 3 of category=64
Sports\Outdoor Recreations
800023 capt stanman and fishing
800036 dinghies
800177 sports outdoor recreations

Found: 3 of category=67
Sports\Tennis
800053 french open
800096 rafael nadal
800175 sports tennis

Candidates for manual augmentation

The following categories are going to be augmented

• Category 5
  
• Category 6
• Category 14
• Category 16
• Category 17
• Category 29
• Category 37
• Category 38
• Category 47
• Category 48
• Category 49
• Category 57
• Category 58
• Category 60
• Category 61
• Category 62
• Category 64
• Category 67
  

Unaugmented Results Info

Parameters:

• svm_learn -d 1 -j 4 -t 1 -i 1

F-1 Score ranking:

• 27 out of 37

Unaugmented SVM Results

Category	TrainingCt	%	Valdn Count	Precision	Recall	Raw Avg	Avg * Count
1	6	1.24%	23	33.33%	4.35%	0.08	1.77
2	10	2.06%	59	45.45%	8.47%	0.14	8.42
3	4	0.82%	7	40.00%	57.14%	0.47	3.29
4	5	1.03%	51	20.00%	1.96%	0.04	1.82
5	4	0.82%	25	0.00%	0.00%	0.00%	0
6	4	0.82%	20	0.00%	0.00%	0.00%	0
7	5	1.03%	11	50.00%	18.18%	0.27	2.93
8	5	1.03%	60	83.33%	8.33%	0.15	9.09
9	9	1.86%	85	33.33%	2.35%	0.04	3.73
10	6	1.24%	71	75.00%	4.23%	0.08	5.69
11	3	0.62%	46	50.00%	2.17%	0.04	1.91
12	6	1.24%	60	66.67%	10.00%	0.17	10.43
13	6	1.24%	69	20.00%	1.45%	0.03	1.87
14	5	1.03%	75	0.00%	0.00%	0.00%	0
15	4	0.82%	55	100.00%	14.55%	0.25	13.97
16	3	0.62%	5	0.00%	0.00%	0.00%	0
17	4	0.82%	34	0.00%	0.00%	0.00%	0
18	38	7.84%	241	37.82%	30.29%	0.34	81.07
19	41	8.45%	399	58.73%	37.09%	0.45	181.41
20	10	2.06%	116	78.57%	9.48%	0.17	19.63
21	10	2.06%	126	53.85%	5.56%	0.1	12.7
22	20	4.12%	252	61.11%	8.73%	0.15	38.5
23	6	1.24%	77	38.46%	6.49%	0.11	8.55
24	9	1.86%	190	100.00%	4.21%	0.08	15.35
25	15	3.09%	141	26.32%	3.55%	0.06	8.82
26	4	0.82%	91	100.00%	3.30%	0.06	5.81
27	4	0.82%	61	33.33%	1.64%	0.03	1.91
28	6	1.24%	40	33.33%	12.50%	0.18	7.27
29	5	1.03%	25	0.00%	0.00%	0.00%	0
30	9	1.86%	125	29.17%	5.60%	0.09	11.75
31	5	1.03%	80	75.00%	3.75%	0.07	5.71
32	4	0.82%	68	77.78%	10.29%	0.18	12.36
33	8	1.65%	44	46.15%	13.64%	0.21	9.26
34	10	2.06%	70	33.33%	8.57%	0.14	9.54
35	8	1.65%	66	50.00%	6.06%	0.11	7.13
36	4	0.82%	99	75.00%	6.06%	0.11	11.1
37	4	0.82%	13	0.00%	0.00%	0.00%	0
38	2	0.41%	29	0.00%	0.00%	0.00%	0
39	3	0.62%	28	50.00%	3.57%	0.07	1.87
40	4	0.82%	32	53.85%	21.88%	0.31	9.96
41	3	0.62%	25	50.00%	4.00%	0.07	1.85
42	4	0.82%	48	33.33%	2.08%	0.04	1.88
43	12	2.47%	100	63.64%	14.00%	0.23	22.95
44	3	0.62%	7	100.00%	28.57%	0.44	3.11
45	4	0.82%	135	62.50%	3.70%	0.07	9.43
46	9	1.86%	365	66.67%	2.74%	0.05	19.21
47	2	0.41%	11	0.00%	0.00%	0.00%	0
48	5	1.03%	40	0.00%	0.00%	0.00%	0
49	3	0.62%	10	0.00%	0.00%	0.00%	0
50	3	0.62%	16	33.33%	6.25%	0.11	1.68
51	7	1.44%	51	26.67%	7.84%	0.12	6.18
52	24	4.95%	301	46.48%	10.96%	0.18	53.39
53	3	0.62%	11	100.00%	9.09%	0.17	1.83
54	2	0.41%	63	100.00%	1.59%	0.03	1.97
55	36	7.42%	332	49.08%	40.06%	0.44	146.46
56	4	0.82%	6	14.29%	16.67%	0.15	0.92
57	4	0.82%	9	0.00%	0.00%	0.00%	0
58	3	0.62%	5	0.00%	0.00%	0.00%	0
59	4	0.82%	9	50.00%	33.33%	0.4	3.6
60	4	0.82%	2	0.00%	0.00%	0.00%	0
61	4	0.82%	11	0.00%	0.00%	0.00%	0
62	3	0.62%	2	0.00%	0.00%	0.00%	0
63	5	1.03%	52	25.00%	1.92%	0.04	1.85
64	3	0.62%	37	0.00%	0.00%	0.00%	0
65	10	2.06%	9	40.00%	44.44%	0.42	3.79
67	3	0.62%	3	0.00%	0.00%	0.00%	0
	485		4829	39.24%		0.12	0.17