discontinued / server · Files

index.html 238 KB
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278 1279 1280 1281 1282 1283 1284 1285 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1451 1452 1453 1454 1455 1456 1457 1458 1459 1460 1461 1462 1463 1464 1465 1466 1467 1468 1469 1470 1471 1472 1473 1474 1475 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 1488 1489 1490 1491 1492 1493 1494 1495 1496 1497 1498 1499 1500 1501 1502 1503 1504 1505 1506 1507 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520 1521 1522 1523 1524 1525 1526 1527 1528 1529 1530 1531 1532 1533 1534 1535 1536 1537 1538 1539 1540 1541 1542 1543 1544 1545 1546 1547 1548 1549 1550 1551 1552 1553 1554 1555 1556 1557 1558 1559 1560 1561 1562 1563 1564 1565 1566 1567 1568 1569 1570 1571 1572 1573 1574 1575 1576 1577 1578 1579 1580 1581 1582 1583 1584 1585 1586 1587 1588 1589 1590 1591 1592 1593 1594 1595 1596 1597 1598 1599 1600 1601 1602 1603 1604 1605 1606 1607 1608 1609 1610 1611 1612 1613 1614 1615 1616 1617 1618 1619 1620 1621 1622 1623 1624 1625 1626 1627 1628 1629 1630 1631 1632 1633 1634 1635 1636 1637 1638 1639 1640 1641 1642 1643 1644 1645 1646 1647 1648 1649 1650 1651 1652 1653 1654 1655 1656 1657 1658 1659 1660 1661 1662 1663 1664 1665 1666 1667 1668 1669 1670 1671 1672 1673 1674 1675 1676 1677 1678 1679 1680 1681 1682 1683 1684 1685 1686 1687 1688 1689 1690 1691 1692 1693 1694 1695 1696 1697 1698 1699 1700 1701 1702 1703 1704 1705 1706 1707 1708 1709 1710 1711 1712 1713 1714 1715 1716 1717 1718 1719 1720 1721 1722 1723 1724 1725 1726 1727 1728 1729 1730 1731 1732 1733 1734 1735 1736 1737 1738 1739 1740 1741 1742 1743 1744 1745 1746 1747 1748 1749 1750 1751 1752 1753 1754 1755 1756 1757 1758 1759 1760 1761 1762 1763 1764 1765 1766 1767 1768 1769 1770 1771 1772 1773 1774 1775 1776 1777 1778 1779 1780 1781 1782 1783 1784 1785 1786 1787 1788 1789 1790 1791 1792 1793 1794 1795 1796 1797 1798 1799 1800 1801 1802 1803 1804 1805 1806 1807 1808 1809 1810 1811 1812 1813 1814 1815 1816 1817 1818 1819 1820 1821 1822 1823 1824 1825 1826 1827 1828 1829 1830 1831 1832 1833 1834 1835 1836 1837 1838 1839 1840 1841 1842 1843 1844 1845 1846 1847 1848 1849 1850 1851 1852 1853 1854 1855 1856 1857 1858 1859 1860 1861 1862 1863 1864 1865 1866 1867 1868 1869 1870 1871 1872 1873 1874 1875 1876 1877 1878 1879 1880 1881 1882 1883 1884 1885 1886 1887 1888 1889 1890 1891 1892 1893 1894 1895 1896 1897 1898 1899 1900 1901 1902 1903 1904 1905 1906 1907 1908 1909 1910 1911 1912 1913 1914 1915 1916 1917 1918 1919 1920 1921 1922 1923 1924 1925 1926 1927 1928 1929 1930 1931 1932 1933 1934 1935 1936 1937 1938 1939 1940 1941 1942 1943 1944 1945 1946 1947 1948 1949 1950 1951 1952 1953 1954 1955 1956 1957 1958 1959 1960 1961 1962 1963 1964 1965 1966 1967 1968 1969 1970 1971 1972 1973 1974 1975 1976 1977 1978 1979 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 2025 2026 2027 2028 2029 2030 2031 2032 2033 2034 2035 2036 2037 2038 2039 2040 2041 2042 2043 2044 2045 2046 2047 2048 2049 2050 2051 2052 2053 2054 2055 2056 2057 2058 2059 2060 2061 2062 2063 2064 2065 2066 2067 2068 2069 2070 2071 2072 2073 2074 2075 2076 2077 2078 2079 2080 2081 2082 2083 2084 2085 2086 2087 2088 2089 2090 2091 2092 2093 2094 2095 2096 2097 2098 2099 2100 2101 2102 2103 2104 2105 2106 2107 2108 2109 2110 2111 2112 2113 2114 2115 2116 2117 2118 2119 2120 2121 2122 2123 2124 2125 2126 2127 2128 2129 2130 2131 2132 2133 2134 2135 2136 2137 2138 2139 2140 2141 2142 2143 2144 2145 2146 2147 2148 2149 2150 2151 2152 2153 2154 2155 2156 2157 2158 2159 2160 2161 2162 2163 2164 2165 2166 2167 2168 2169 2170 2171 2172 2173 2174 2175 2176 2177 2178 2179 2180 2181 2182 2183 2184 2185 2186 2187 2188 2189 2190 2191 2192 2193 2194 2195 2196 2197 2198 2199 2200 2201 2202 2203 2204 2205 2206 2207 2208 2209 2210 2211 2212 2213 2214 2215 2216 2217 2218 2219 2220 2221 2222 2223 2224 2225 2226 2227 2228 2229 2230 2231 2232 2233 2234 2235 2236 2237 2238 2239 2240 2241 2242 2243 2244 2245 2246 2247 2248 2249 2250 2251 2252 2253 2254 2255 2256 2257 2258 2259 2260 2261 2262 2263 2264 2265 2266 2267 2268 2269 2270 2271 2272 2273 2274 2275 2276 2277 2278 2279 2280 2281 2282 2283 2284 2285 2286 2287 2288 2289 2290 2291 2292 2293 2294 2295 2296 2297 2298 2299 2300 2301 2302 2303 2304 2305 2306 2307 2308 2309 2310 2311 2312 2313 2314 2315 2316 2317 2318 2319 2320 2321 2322 2323 2324 2325 2326 2327 2328 2329 2330 2331 2332 2333 2334 2335 2336 2337 2338 2339 2340 2341 2342 2343 2344 2345 2346 2347 2348 2349 2350 2351 2352 2353 2354 2355 2356 2357 2358 2359 2360 2361 2362 2363 2364 2365 2366 2367 2368 2369 2370 2371 2372 2373 2374 2375 2376 2377 2378 2379 2380 2381 2382 2383 2384 2385 2386 2387 2388 2389 2390 2391 2392 2393 2394 2395 2396 2397 2398 2399 2400 2401 2402 2403 2404 2405 2406 2407 2408 2409 2410 2411 2412 2413 2414 2415 2416 2417 2418 2419 2420 2421 2422 2423 2424 2425 2426 2427 2428 2429 2430 2431 2432 2433 2434 2435 2436 2437 2438 2439 2440 2441 2442 2443 2444 2445 2446 2447 2448 2449 2450 2451 2452 2453 2454 2455 2456 2457 2458 2459 2460 2461 2462 2463 2464 2465 2466 2467 2468 2469 2470 2471 2472 2473 2474 2475 2476 2477 2478 2479 2480 2481 2482 2483 2484 2485 2486 2487 2488 2489 2490 2491 2492 2493 2494 2495 2496 2497 2498 2499 2500 2501 2502 2503 2504 2505 2506 2507 2508 2509 2510 2511 2512 2513 2514 2515 2516 2517 2518 2519 2520 2521 2522 2523 2524 2525 2526 2527 2528 2529 2530 2531 2532 2533 2534 2535 2536 2537 2538 2539 2540 2541 2542 2543 2544 2545 2546 2547 2548 2549 2550 2551 2552 2553 2554 2555 2556 2557 2558 2559 2560 2561 2562 2563 2564 2565 2566 2567 2568 2569 2570 2571 2572 2573 2574 2575 2576 2577 2578 2579 2580 2581 2582 2583 2584 2585 2586 2587 2588 2589 2590 2591 2592 2593 2594 2595 2596 2597 2598 2599 2600 2601 2602 2603 2604 2605 2606 2607 2608 2609 2610 2611 2612 2613 2614 2615 2616 2617 2618 2619 2620 2621 2622 2623 2624 2625 2626 2627 2628 2629 2630 2631 2632 2633 2634 2635 2636 2637 2638 2639 2640 2641 2642 2643 2644 2645 2646 2647 2648 2649 2650 2651 2652 2653 2654 2655 2656 2657 2658 2659 2660 2661 2662 2663 2664 2665 2666 2667 2668 2669 2670 2671 2672 2673 2674 2675 2676 2677 2678 2679 2680 2681 2682 2683 2684 2685 2686 2687 2688 2689 2690 2691 2692 2693 2694 2695 2696 2697 2698 2699 2700 2701 2702 2703 2704 2705 2706 2707 2708 2709 2710 2711 2712 2713 2714 2715 2716 2717 2718 2719 2720 2721 2722 2723 2724 2725 2726 2727 2728 2729 2730 2731 2732 2733 2734 2735 2736 2737 2738 2739 2740 2741 2742 2743 2744 2745 2746 2747 2748 2749 2750 2751 2752 2753 2754 2755 2756 2757 2758 2759 2760 2761 2762 2763 2764 2765 2766 2767 2768 2769 2770 2771 2772 2773 2774 2775 2776 2777 2778 2779 2780 2781 2782 2783 2784 2785 2786 2787 2788 2789 2790 2791 2792 2793 2794 2795 2796 2797 2798 2799 2800 2801 2802 2803 2804 2805 2806 2807 2808 2809 2810 2811 2812 2813 2814 2815 2816 2817 2818 2819 2820 2821 2822 2823 2824 2825 2826 2827 2828 2829 2830 2831 2832 2833 2834 2835 2836 2837 2838 2839 2840 2841 2842 2843 2844 2845 2846 2847 2848 2849 2850 2851 2852 2853 2854 2855 2856 2857 2858 2859 2860 2861 2862 2863 2864 2865 2866 2867 2868 2869 2870 2871 2872 2873 2874 2875 2876 2877 2878 2879 2880 2881 2882 2883 2884 2885 2886 2887 2888 2889 2890 2891 2892 2893 2894 2895 2896 2897 2898 2899 2900 2901 2902 2903 2904 2905 2906 2907 2908 2909 2910 2911 2912 2913 2914 2915 2916 2917 2918 2919 2920 2921 2922 2923 2924 2925 2926 2927 2928 2929 2930 2931 2932 2933 2934 2935 2936 2937 2938 2939 2940 2941 2942 2943 2944 2945 2946 2947 2948 2949 2950 2951 2952 2953 2954 2955 2956 2957 2958 2959 2960 2961 2962 2963 2964 2965 2966 2967 2968 2969 2970 2971 2972 2973 2974 2975 2976 2977 2978 2979 2980 2981 2982 2983 2984 2985 2986 2987 2988 2989 2990 2991 2992 2993 2994 2995 2996 2997 2998 2999 3000 3001 3002 3003 3004 3005 3006 3007 3008 3009 3010 3011 3012 3013 3014 3015 3016 3017 3018 3019 3020 3021 3022 3023 3024 3025 3026 3027 3028 3029 3030 3031 3032 3033 3034 3035 3036 3037 3038 3039 3040 3041 3042 3043 3044 3045 3046 3047 3048 3049 3050 3051 3052 3053 3054 3055 3056 3057 3058 3059 3060 3061 3062 3063 3064 3065 3066 3067 3068 3069 3070 3071 3072 3073 3074 3075 3076 3077 3078 3079 3080 3081 3082 3083 3084 3085 3086 3087 3088 3089 3090 3091 3092 3093 3094 3095 3096 3097 3098 3099 3100 3101 3102 3103 3104 3105 3106 3107 3108 3109 3110 3111 3112 3113 3114 3115 3116 3117 3118 3119 3120 3121 3122 3123 3124 3125 3126 3127 3128 3129 3130 3131 3132 3133 3134 3135 3136 3137 3138 3139 3140 3141 3142 3143 3144 3145 3146 3147 3148 3149 3150 3151 3152 3153 3154 3155 3156 3157 3158 3159 3160 3161 3162 3163 3164 3165 3166 3167 3168 3169 3170 3171 3172 3173 3174 3175 3176 3177 3178 3179 3180 3181 3182 3183 3184 3185 3186 3187 3188 3189 3190 3191 3192 3193 3194 3195 3196 3197 3198 3199 3200 3201 3202 3203 3204 3205 3206 3207 3208 3209 3210 3211 3212 3213 3214 3215 3216 3217 3218 3219 3220 3221 3222 3223 3224 3225 3226 3227 3228 3229 3230 3231 3232 3233 3234 3235 3236 3237 3238 3239 3240 3241 3242 3243 3244 3245 3246 3247 3248 3249 3250 3251 3252 3253 3254 3255 3256 3257 3258 3259 3260 3261 3262 3263 3264 3265 3266 3267 3268 3269 3270 3271 3272 3273 3274 3275 3276 3277 3278 3279 3280 3281 3282 3283 3284 3285 3286 3287 3288 3289 3290 3291 3292 3293 3294 3295 3296 3297 3298 3299 3300 3301 3302 3303 3304 3305 3306 3307 3308 3309 3310 3311 3312 3313 3314 3315 3316 3317 3318 3319 3320 3321 3322 3323 3324 3325 3326 3327 3328 3329 3330 3331 3332 3333 3334 3335 3336 3337 3338 3339 3340 3341 3342 3343 3344 3345 3346 3347 3348 3349 3350 3351 3352 3353 3354 3355 3356 3357 3358 3359 3360 3361 3362 3363 3364 3365 3366 3367 3368 3369 3370 3371 3372 3373 3374 3375 3376 3377 3378 3379 3380 3381 3382 3383 3384 3385 3386 3387 3388 3389 3390 3391 3392 3393 3394 3395 3396 3397 3398 3399 3400 3401 3402 3403 3404 3405 3406 3407 3408 3409 3410 3411 3412 3413 3414 3415 3416 3417 3418 3419 3420 3421 3422 3423 3424 3425 3426 3427 3428 3429 3430 3431 3432 3433 3434 3435 3436 3437 3438 3439 3440 3441 3442 3443 3444 3445 3446 3447 3448 3449 3450 3451 3452 3453 3454 3455 3456 3457 3458 3459 3460 3461 3462 3463 3464 3465 3466 3467 3468 3469 3470 3471 3472 3473 3474 3475 3476 3477 3478 3479 3480 3481 3482 3483 3484 3485 3486 3487 3488 3489 3490 3491 3492 3493 3494 3495 3496 3497 3498 3499 3500 3501 3502 3503 3504 3505 3506 3507 3508 3509 3510 3511 3512 3513 3514 3515 3516 3517 3518 3519 3520 3521 3522 3523 3524 3525 3526 3527 3528 3529 3530 3531 3532 3533 3534 3535 3536 3537 3538 3539 3540 3541 3542 3543 3544 3545 3546 3547 3548 3549 3550 3551 3552 3553 3554 3555 3556 3557 3558 3559 3560 3561 3562 3563 3564 3565 3566 3567 3568 3569 3570 3571 3572 3573 3574 3575 3576 3577 3578 3579 3580 3581 3582 3583 3584 3585 3586 3587 3588 3589 3590 3591 3592 3593 3594 3595 3596 3597 3598 3599 3600 3601 3602 3603 3604 3605 3606 3607 3608 3609 3610 3611 3612 3613 3614 3615 3616 3617 3618 3619 3620 3621 3622 3623 3624 3625 3626 3627 3628 3629 3630 3631 3632 3633 3634 3635 3636 3637 3638 3639 3640 3641 3642 3643 3644 3645 3646 3647 3648 3649 3650 3651 3652 3653 3654 3655 3656 3657 3658 3659 3660 3661 3662 3663 3664 3665 3666 3667 3668 3669 3670 3671 3672 3673 3674 3675 3676 3677 3678 3679 3680 3681 3682 3683 3684 3685 3686 3687 3688 3689 3690 3691 3692 3693 3694 3695 3696 3697 3698 3699 3700 3701 3702 3703 3704 3705 3706 3707 3708 3709 3710 3711 3712 3713 3714 3715 3716 3717 3718 3719 3720 3721 3722 3723 3724 3725 3726 3727 3728 3729 3730 3731 3732 3733 3734 3735 3736 3737 3738 3739 3740 3741 3742 3743 3744 3745 3746 3747 3748 3749 3750 3751 3752 3753 3754 3755 3756 3757 3758 3759 3760 3761 3762 3763 3764 3765 3766 3767 3768 3769 3770 3771 3772 3773 3774 3775 3776 3777 3778 3779 3780 3781 3782 3783 3784 3785 3786 3787 3788 3789 3790 3791 3792 3793 3794 3795 3796 3797 3798 3799 3800 3801 3802 3803 3804 3805 3806 3807 3808 3809 3810 3811 3812 3813 3814 3815 3816 3817 3818 3819 3820 3821 3822 3823 3824 3825 3826 3827 3828 3829 3830 3831 3832 3833 3834 3835 3836 3837 3838 3839 3840 3841 3842 3843 3844 3845 3846 3847 3848 3849 3850 3851 3852 3853 3854 3855 3856 3857 3858 3859 3860 3861 3862 3863 3864 3865 3866 3867 3868 3869 3870 3871 3872 3873 3874 3875 3876 3877 3878 3879 3880 3881 3882 3883 3884 3885 3886 3887 3888 3889 3890 3891 3892 3893 3894 3895 3896 3897 3898 3899 3900 3901 3902 3903 3904 3905 3906 3907 3908 3909 3910 3911 3912 3913 3914 3915 3916 3917 3918 3919 3920 3921 3922 3923 3924 3925 3926 3927 3928 3929 3930 3931 3932 3933 3934 3935 3936 3937 3938 3939 3940 3941 3942 3943 3944 3945 3946 3947 3948 3949 3950 3951 3952 3953 3954 3955 3956 3957 3958 3959 3960 3961 3962 3963 3964 3965 3966 3967 3968 3969 3970 3971 3972 3973 3974 3975 3976 3977 3978 3979 3980 3981 3982 3983 3984 3985 3986 3987 3988 3989 3990 3991 3992 3993 3994 3995 3996 3997 3998 3999 4000 4001 4002 4003 4004 4005 4006 4007 4008 4009 4010 4011 4012 4013 4014 4015 4016 4017 4018 4019 4020 4021 4022 4023 4024 4025 4026 4027 4028 4029 4030 4031 4032 4033 4034 4035 4036 4037 4038 4039 4040 4041 4042 4043 4044 4045 4046 4047 4048 4049 4050 4051 4052 4053 4054 4055 4056 4057 4058 4059 4060 4061 4062 4063 4064 4065 4066 4067 4068 4069 4070 4071 4072 4073 4074 4075 4076 4077 4078 4079 4080 4081 4082 4083 4084 4085 4086 4087 4088 4089 4090 4091 4092 4093 4094 4095 4096 4097 4098 4099 4100 4101 4102 4103 4104 4105 4106 4107 4108 4109 4110 4111 4112 4113 4114 4115 4116 4117 4118 4119 4120 4121 4122 4123 4124 4125 4126 4127 4128 4129 4130 4131 4132 4133 4134 4135 4136 4137 4138 4139 4140 4141 4142 4143 4144 4145 4146 4147 4148 4149 4150 4151 4152 4153 4154 4155 4156 4157 4158 4159 4160 4161 4162 4163 4164 4165 4166 4167 4168 4169 4170 4171 4172 4173 4174 4175 4176 4177 4178 4179 4180 4181 4182 4183 4184 4185 4186 4187 4188 4189 4190 4191 4192 4193 4194 4195 4196 4197 4198 4199 4200 4201 4202 4203 4204 4205 4206 4207 4208 4209 4210 4211 4212 4213 4214 4215 4216 4217 4218 4219 4220 4221 4222 4223 4224 4225 4226 4227 4228 4229 4230 4231 4232 4233 4234 4235 4236 4237 4238 4239 4240 4241 4242 4243 4244 4245 4246 4247 4248 4249 4250 4251 4252 4253 4254 4255 4256 4257 4258 4259 4260 4261 4262 4263 4264 4265 4266 4267 4268 4269 4270 4271 4272 4273 4274 4275 4276 4277 4278 4279 4280 4281 4282 4283 4284 4285 4286 4287 4288 4289 4290 4291 4292 4293 4294 4295 4296 4297 4298 4299 4300 4301 4302 4303 4304 4305 4306 4307 4308 4309 4310 4311 4312 4313 4314 4315 4316 4317 4318 4319 4320 4321 4322 4323 4324 4325 4326 4327 4328 4329 4330 4331 4332 4333 4334 4335 4336 4337 4338 4339 4340 4341 4342 4343 4344 4345 4346 4347 4348 4349 4350 4351 4352 4353 4354 4355 4356 4357 4358 4359 4360 4361 4362 4363 4364 4365 4366 4367 4368 4369 4370 4371 4372 4373 4374 4375 4376 4377 4378 4379 4380 4381 4382 4383 4384 4385 4386 4387 4388 4389 4390 4391 4392 4393 4394 4395 4396 4397 4398 4399 4400 4401 4402 4403 4404 4405 4406 4407 4408 4409 4410 4411 4412 4413 4414 4415 4416 4417 4418 4419 4420 4421 4422 4423 4424 4425 4426 4427 4428 4429 4430 4431 4432 4433 4434 4435 4436 4437 4438 4439 4440 4441 4442 4443 4444 4445 4446 4447 4448 4449 4450 4451 4452 4453 4454 4455 4456 4457 4458 4459 4460 4461 4462 4463 4464 4465 4466 4467 4468 4469 4470 4471 4472 4473 4474 4475 4476 4477 4478 4479 4480 4481 4482 4483 4484 4485 4486 4487 4488 4489 4490 4491 4492 4493 4494 4495 4496 4497 4498 4499 4500 4501 4502 4503 4504 4505 4506 4507 4508 4509 4510 4511 4512 4513 4514 4515 4516 4517 4518 4519 4520 4521 4522 4523 4524 4525 4526 4527 4528 4529 4530 4531 4532 4533 4534 4535 4536 4537 4538 4539 4540 4541 4542 4543 4544 4545 4546 4547 4548 4549 4550 4551 4552 4553 4554 4555 4556 4557 4558 4559 4560 4561 4562 4563 4564 4565 4566 4567 4568 4569 4570 4571 4572 4573 4574 4575 4576 4577 4578 4579 4580 4581 4582 4583 4584 4585 4586 4587 4588 4589 4590 4591 4592 4593 4594 4595 4596 4597 4598 4599 4600 4601 4602 4603 4604 4605 4606 4607 4608 4609 4610 4611 4612 4613 4614 4615 4616 4617 4618 4619 4620 4621 4622 4623 4624 4625 4626 4627 4628 4629 4630 4631 4632 4633 4634 4635 4636 4637 4638 4639 4640 4641 4642 4643 4644 4645 4646 4647 4648 4649 4650 4651 4652 4653 4654 4655 4656 4657 4658 4659 4660 4661 4662 4663 4664 4665 4666 4667 4668 4669 4670 4671 4672 4673 4674 4675 4676 4677 4678 4679 4680 4681 4682 4683 4684 4685 4686 4687 4688 4689 4690 4691 4692 4693 4694 4695 4696 4697 4698 4699 4700 4701 4702 4703 4704 4705 4706 4707 4708 4709 4710 4711 4712 4713 4714 4715 4716 4717 4718 4719 4720 4721 4722 4723 4724 4725 4726 4727 4728 4729 4730 4731 4732 4733 4734 4735 4736 4737 4738 4739 4740 4741 4742 4743 4744 4745 4746 4747 4748 4749 4750 4751 4752 4753 4754 4755 4756 4757 4758 4759 4760 4761 4762 4763 4764 4765 4766 4767 4768 4769 4770 4771 4772 4773 4774 4775 4776 4777 4778 4779 4780 4781 4782 4783 4784 4785 4786 4787 4788 4789 4790 4791 4792 4793 4794 4795 4796 4797 4798 4799 4800 4801 4802 4803 4804 4805 4806 4807 4808 4809 4810 4811 4812 4813 4814 4815 4816 4817 4818 4819 4820 4821 4822 4823 4824 4825 4826 4827 4828 4829 4830 4831 4832 4833 4834 4835 4836 4837 4838 4839 4840 4841 4842 4843 4844 4845 4846 4847 4848 4849 4850 4851 4852 4853 4854 4855 4856 4857 4858 4859 4860 4861 4862 4863 4864 4865 4866 4867 4868 4869 4870 4871 4872 4873 4874 4875 4876 4877 4878 4879 4880 4881 4882 4883 4884 4885 4886 4887 4888 4889 4890 4891 4892 4893 4894 4895 4896 4897 4898 4899 4900 4901 4902 4903 4904 4905 4906 4907 4908 4909 4910 4911 4912 4913 4914 4915 4916 4917 4918 4919 4920 4921 4922 4923 4924 4925 4926 4927 4928 4929 4930 4931 4932 4933 4934 4935 4936 4937 4938 4939 4940 4941 4942 4943 4944 4945 4946 4947 4948 4949 4950 4951 4952 4953 4954 4955 4956 4957 4958 4959 4960 4961 4962 4963 4964 4965 4966 4967 4968 4969 4970 4971 4972 4973 4974 4975 4976 4977 4978 4979 4980 4981 4982 4983 4984 4985 4986 4987 4988 4989 4990 4991 4992 4993 4994 4995 4996 4997 4998 4999 5000 5001 5002 5003 5004 5005 5006 5007 5008 5009 5010 5011 5012 5013 5014 5015 5016 5017 5018 5019 5020 5021 5022 5023 5024 5025 5026 5027 5028 5029 5030 5031 5032 5033 5034 5035 5036 5037 5038 5039 5040 5041 5042 5043 5044 5045 5046 5047 5048 5049 5050 5051 5052 5053 5054 5055 5056 5057 5058 5059 5060 5061 5062 5063 5064 5065 5066 5067 5068 5069 5070 5071 5072 5073 5074 5075 5076 5077 5078 5079 5080 5081 5082 5083 5084 5085 5086 5087 5088 5089 5090 5091 5092 5093 5094 5095 5096 5097 5098 5099 5100 5101 5102 5103 5104 5105 5106 5107 5108 5109 5110 5111 5112 5113 5114 5115 5116 5117 5118 5119 5120 5121 5122 5123 5124 5125 5126 5127 5128 5129 5130 5131 5132 5133 5134 5135 5136 5137 5138 5139 5140 5141 5142 5143 5144 5145 5146 5147 5148 5149 5150 5151 5152 5153 5154 5155 5156 5157 5158 5159 5160 5161 5162 5163 5164 5165 5166 5167 5168 5169 5170 5171 5172 5173 5174 5175 5176 5177 5178 5179 5180 5181 5182 5183 5184 5185 5186 5187 5188 5189 5190 5191 5192 5193 5194 5195 5196 5197 5198 5199 5200 5201 5202 5203 5204 5205 5206 5207 5208 5209 5210 5211 5212 5213 5214 5215 5216 5217 5218 5219 5220 5221 5222 5223 5224 5225 5226 5227 5228 5229 5230 5231 5232 5233 5234 5235 5236 5237 5238 5239 5240 5241 5242 5243 5244 5245 5246 5247 5248 5249 5250 5251 5252 5253 5254 5255 5256 5257 5258 5259 5260 5261 5262 5263 5264 5265 5266 5267 5268 5269 5270 5271 5272 5273 5274 5275 5276 5277 5278 5279 5280 5281 5282 5283 5284 5285 5286 5287 5288 5289 5290 5291 5292 5293 5294 5295 5296 5297 5298 5299 5300 5301 5302 5303 5304 5305 5306 5307 5308 5309 5310 5311 5312 5313 5314 5315 5316 5317 5318 5319 5320 5321 5322 5323 5324 5325 5326 5327 5328 5329 5330 5331 5332 5333 5334 5335 5336 5337 5338 5339 5340 5341 5342 5343 5344 5345 5346 5347 5348 5349 5350 5351 5352 5353 5354 5355 5356 5357 5358 5359 5360 5361 5362 5363 5364 5365 5366 5367 5368 5369 5370 5371 5372 5373 5374 5375 5376 5377 5378 5379 5380 5381 5382 5383 5384 5385 5386 5387 5388 5389 5390 5391 5392 5393 5394 5395 5396 5397 5398 5399 5400 5401 5402 5403 5404 5405 5406 5407 5408 5409 5410 5411 5412 5413 5414 5415 5416 5417 5418 5419 5420 5421 5422 5423 5424 5425 5426 5427 5428 5429 5430 5431 5432 5433 5434 5435 5436 5437 5438 5439 5440 5441 5442 5443 5444 5445 5446 5447 5448 5449 5450 5451 5452 5453 5454 5455 5456 5457 5458 5459 5460 5461 5462 5463 5464 5465 5466 5467 5468 5469 5470 5471 5472 5473 5474 5475 5476 5477 5478 5479 5480 5481 5482 5483 5484 5485 5486 5487 5488 5489 5490 5491 5492 5493 5494 5495 5496 5497 5498 5499 5500 5501 5502 5503 5504 5505 5506 5507 5508 5509 5510 5511 5512 5513 5514 5515 5516 5517 5518 5519 5520 5521 5522 5523 5524 5525 5526 5527 5528 5529 5530 5531 5532 5533 5534 5535 5536 5537 5538 5539 5540 5541 5542 5543 5544 5545 5546 5547 5548 5549 5550 5551 5552 5553 5554 5555 5556 5557 5558 5559 5560 5561 5562 5563 5564 5565 5566 5567 5568 5569 5570 5571 5572 5573 5574 5575 5576 5577 5578 5579 5580 5581 5582 5583 5584 5585 5586 5587 5588 5589 5590 5591 5592 5593 5594 5595 5596 5597 5598 5599 5600 5601 5602 5603 5604 5605 5606 5607 5608 5609 5610 5611 5612 5613 5614 5615 5616 5617 5618 5619 5620 5621 5622 5623 5624 5625 5626 5627 5628 5629 5630 5631 5632 5633 5634 5635 5636 5637 5638 5639 5640 5641 5642 5643 5644 5645 5646 5647 5648 5649 5650 5651 5652 5653 5654 5655 5656 5657 5658 5659 5660 5661 5662 5663 5664 5665 5666 5667 5668 5669 5670 5671 5672 5673 5674 5675 5676 5677 5678 5679 5680 5681 5682 5683 5684 5685 5686 5687 5688 5689 5690 5691 5692 5693 5694 5695 5696 5697 5698 5699 5700 5701 5702 5703 5704 5705 5706 5707 5708 5709 5710 5711 5712 5713 5714 5715 5716 5717 5718 5719 5720 5721 5722 5723 5724 5725 5726 5727 5728 5729 5730 5731 5732 5733 5734 5735 5736 5737 5738 5739 5740 5741 5742 5743 5744 5745 5746 5747 5748 5749 5750 5751 5752 5753 5754 5755 5756 5757 5758 5759 5760 5761 5762 5763 5764 5765 5766 5767 5768 5769 5770 5771 5772 5773 5774 5775 5776 5777 5778 5779 5780 5781 5782 5783 5784 5785 5786 5787 5788 5789 5790 5791 5792 5793 5794 5795 5796 5797 5798 5799 5800 5801 5802 5803 5804 5805 5806 5807 5808 5809 5810 5811 5812 5813 5814 5815 5816 5817 5818 5819 5820 5821 5822 5823 5824 5825 5826 5827 5828 5829 5830 5831 5832 5833 5834 5835 5836 5837 5838 5839 5840 5841 5842 5843 5844 5845 5846 5847 5848 5849 5850 5851 5852 5853 5854 5855 5856 5857 5858 5859 5860 5861 5862 5863 5864 5865 5866 5867 5868 5869 5870 5871 5872 5873 5874 5875 5876 5877 5878 5879 5880 5881 5882 5883 5884 5885 5886 5887 5888 5889 5890 5891 5892 5893 5894 5895 5896 5897 5898 5899 5900 5901 5902 5903 5904 5905 5906 5907 5908 5909 5910 5911 5912 5913 5914 5915 5916 5917 5918 5919 5920 5921 5922 5923 5924 5925 5926 5927 5928 5929 5930 5931 5932 5933 5934 5935 5936 5937 5938 5939 5940 5941 5942 5943 5944 5945 5946 
<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta name="generator" content=
"HTML Tidy for Linux/x86 (vers 1 September 2005), see www.w3.org" />
<meta http-equiv="Content-Type" content= "text/html; charset=utf-8" />
<title>EMMA: Extensible MultiModal Annotation markup
language</title>

<style type="text/css">
/*<![CDATA[*/
span.term {
  color: rgb(0,0,192);
  font-style: italic
  }
blockquote { margin-left: 4% }
.toc { list-style-type: none; marker-offset: 1em }
.tocline { list-style-type: none }
ul.toc a { text-decoration: none }
.fig { text-align: center }
pre { font-family: monospace }
pre.example {
  margin-left: 0;
  padding: 0.5em;
  width: 98%;
  font-family: monospace;
  white-space: pre;
  border: none;
  font-size: 95%;
  background-color: rgb(230,230,255);
  }
.note { color: red }
.new { color: green;}
.old { text-decoration: line-through }
.newer { text-decoration: underline }
.change { color: red }
.changeTable { color: orange }
.remove { text-decoration: line-through }
div.issues {
  border-width: thin;
  border-style: solid;
  border-color: maroon;
  background-color: #FFEECC;
  color: maroon;
  width: 95%; padding: 0.5em; }
div.issues h4 { margin-top: 0 }
code {
  font-weight:bold;
  color: green;
  font-family: monospace;
  font-size: 110%;
  }
.good {
  border: green 2px solid;
  font-weight: bold;
  color: green;
  margin: 1em 5% 1em 0px;
  }
.bad {
  border: red 2px solid;
  font-weight: bold;
  color: rgb(192,101,101);
  margin: 1em 5% 1em 0px;
  }
div.navbar { text-align: center }
div.contents {
  border: medium none;
  padding: 0.5em;
  margin-right: 5%;
  background-color: rgb(230,230,255);
  }
table.exceptions {
  background-color: rgb(255,255,153)
  }
table.modes { font-size: 90% }
table.defn {
  border-width: thin;
  border-style: solid;
  border-color: black;
  color: black
  }
table.defn th { background-color: rgb(220,220,255);
  border-style: solid; border-color: black; border-width: thin }
table.defn td { background-color: rgb(230,230,255);
  border-style: solid; border-color: black; border-width: thin }
.diff { color: rgb(128,0,0) }
.reqs {  color: blue; font-style: italic  }
.editorial { color: maroon; font-style: italic }
/*]]>*/
</style>
<link rel="stylesheet" type="text/css" href=
"http://www.w3.org/StyleSheets/TR/W3C-REC.css" />
</head>
<body>
<div class="head">
<div class="banner"><a href="http://www.w3.org/"><img alt="W3C"
src="http://www.w3.org/Icons/w3c_home" width="72" height=
"48" /></a></div>
<h1 class="notoc" id="s0">EMMA: Extensible MultiModal Annotation
markup language</h1>
<h2><a id="w3c-doctype" name="w3c-doctype"><acronym title=
"World Wide Web Consortium">W3C</acronym> Recommendation
10 February 2009</a></h2>
<dl>
<dt>This version:</dt>
<dd><a href=
"http://www.w3.org/TR/2009/REC-emma-20090210/">http://www.w3.org/TR/2009/REC-emma-20090210/</a></dd>
<dt>Latest version:</dt>
<dd><a href=
"http://www.w3.org/TR/emma/">http://www.w3.org/TR/emma/</a></dd>
<dt>Previous version:</dt>
<dd><a href=
"http://www.w3.org/TR/2008/PR-emma-20081215/">http://www.w3.org/TR/2008/PR-emma-20081215/</a></dd>
</dl>
<dl>
<dt>Editor:</dt>
<dd>Michael Johnston, AT&amp;T</dd>
<dt>Authors:</dt>
<dd>Paolo Baggia, Loquendo</dd>
<dd>Daniel C. Burnett, Voxeo (formerly of Vocalocity and Nuance)</dd>
<dd>Jerry Carter, Nuance</dd>
<dd>Deborah A. Dahl, Invited Expert</dd>
<dd>Gerry McCobb, Openstream</dd>
<dd>Dave Raggett, (until 2007, while at W3C/Volantis and W3C/Canon)</dd>
</dl>

    <p>Please refer to the
    <a href="http://www.w3.org/2009/02/emma-errata.html">
    <strong>errata</strong></a>
    for this document, which may include some normative
    corrections.</p>

    <p>See also
    <a href="http://www.w3.org/2003/03/Translations/byTechnology?technology=emma">
    <strong>translations</strong></a>.</p>

<p class="copyright"><a href="http://www.w3.org/Consortium/Legal/ipr-notice#Copyright">Copyright</a> &copy; 2009 <a href="http://www.w3.org/"><acronym title="World Wide Web Consortium">W3C</acronym></a><sup>&reg;</sup> (<a href="http://www.csail.mit.edu/"><acronym title="Massachusetts Institute of Technology">MIT</acronym></a>, <a href="http://www.ercim.org/"><acronym title="European Research Consortium for Informatics and Mathematics">ERCIM</acronym></a>, <a href="http://www.keio.ac.jp/">Keio</a>), All Rights Reserved. W3C <a href="http://www.w3.org/Consortium/Legal/ipr-notice#Legal_Disclaimer">liability</a>, <a href="http://www.w3.org/Consortium/Legal/ipr-notice#W3C_Trademarks">trademark</a> and <a href="http://www.w3.org/Consortium/Legal/copyright-documents">document use</a> rules apply.</p>

<hr title="Separator for header" /></div>
<h2 class="notoc" id="abstract">Abstract</h2>
<p>The W3C Multimodal Interaction Working Group aims to develop
specifications to enable access to the Web using multimodal
interaction. This document is part of a set of specifications for
multimodal systems, and provides details of an XML markup language
for containing and annotating the interpretation of user input.
Examples of interpretation of user input are a transcription into
words of a raw signal, for instance derived from speech, pen or
keystroke input, a set of attribute/value pairs describing their
meaning, or a set of attribute/value pairs describing a gesture.
The interpretation of the user's input is expected to be generated
by signal interpretation processes, such as speech and ink
recognition, semantic interpreters, and other types of processors
for use by components that act on the user's inputs such as
interaction managers.</p>
<h2 id="status">Status of this Document</h2>
<p><em>This section describes the status of this document at the
time of its publication. Other documents may supersede this
document. A list of current W3C publications and the latest
revision of this technical report can be found in the <a href=
"http://www.w3.org/TR/">W3C technical reports index</a> at
http://www.w3.org/TR/.</em></p>

<p>This is the
<a href="http://www.w3.org/2005/10/Process-20051014/tr.html#RecsW3C">
Recommendation
</a>
of "EMMA: Extensible MultiModal Annotation markup language".

It has been produced by the
<a href="http://www.w3.org/2002/mmi/">Multimodal Interaction Working Group</a>,
which is part of the
<a href="http://www.w3.org/2002/mmi/Activity.html">Multimodal Interaction Activity</a>.
</p>

<p>Comments are welcome on <a href="mailto:www-multimodal@w3.org">www-multimodal@w3.org</a>
(<a href="http://lists.w3.org/Archives/Public/www-multimodal/">archive</a>).

See <a href="http://www.w3.org/Mail/">W3C mailing list and archive
usage guidelines</a>.</p>

<p>The design of EMMA has been widely reviewed
(see the <a href="http://www.w3.org/TR/2008/PR-emma-20081215/emma-disp.html">
disposition of comments</a>)
and satisfies the Working Group's technical requirements.

A list of implementations is included in the
<a href="http://www.w3.org/2002/mmi/2008/emma-ir/">
EMMA Implementation Report</a>.

The Working Group made a few editorial changes to the
<a href="http://www.w3.org/TR/2008/PR-emma-20081215/">
15 December 2008 Proposed Recommendation</a>.
Changes from the Proposed Recommendation can be found in
<a href="#appF">Appendix F</a>.
</p>


<p>This document has been reviewed by W3C Members, by software
  developers, and by other W3C groups and interested parties, and is
  endorsed by the Director as a W3C Recommendation. It is a stable
  document and may be used as reference material or cited from another
  document. W3C's role in making the Recommendation is to draw
  attention to the specification and to promote its widespread
  deployment. This enhances the functionality and interoperability of
  the Web.</p>

<p>This specification describes markup for representing
interpretations of user input (speech, keystrokes, pen input etc.)
together with annotations for confidence scores, timestamps, input
medium etc., and forms part of the proposals for the <a href=
"http://www.w3.org/TR/mmi-framework/">W3C Multimodal Interaction
Framework</a>.</p>

<p>This document was produced by a group operating under the
<a href="http://www.w3.org/Consortium/Patent-Policy-20040205/">5
February 2004 W3C Patent Policy</a>. W3C maintains a <a rel=
"disclosure" href=
"http://www.w3.org/2004/01/pp-impl/34607/status">public list of any
patent disclosures</a> made in connection with the deliverables of
the group; that page also includes instructions for disclosing a
patent. An individual who has actual knowledge of a patent which
the individual believes contains <a href=
"http://www.w3.org/Consortium/Patent-Policy-20040205/#def-essential">
Essential Claim(s)</a> must disclose the information in accordance
with <a href=
"http://www.w3.org/Consortium/Patent-Policy-20040205/#sec-Disclosure">
section 6 of the W3C Patent Policy</a>.</p>

<p>The sections in the main body of this document are normative unless
otherwise specified.  The appendices in this document are informative
unless otherwise indicated explicitly.</p>


<h2 class="notoc" id="conv">Conventions of this Document</h2>
<p>All sections in this specification are normative, unless
otherwise indicated. The informative parts of this specification
are identified by "Informative" labels within sections.</p>
<p>The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL
NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL"
in this document are to be interpreted as described in [<a href=
"#ref-rfc2119">RFC2119</a>].</p>
<h2 class="notoc" id="toc">Table of Contents</h2>
<ul class="tocline">
<li>1. <a href="#s1">Introduction</a>
<ul class="tocline">
<li>1.1 <a href="#s1.1">Uses of EMMA</a></li>
<li>1.2 <a href="#s1.2">Terminology</a></li>
</ul>
</li>
<li>2. <a href="#s2">Structure of EMMA documents</a>
<ul class="tocline">
<li>2.<span>1</span> <a href="#s2.1">Data model</a></li>
<li>2.<span>2</span> <a href="#s2.2">EMMA namespace
prefixes</a></li>
</ul>
</li>
<li>3. <a href="#s3">EMMA structural elements</a>
<ul class="tocline">
<li>3.1 <a href="#s3.1">Root element:
<code>emma:emma</code></a></li>
<li>3.2 <a href="#s3.2">Interpretation element:
<code>emma:interpretation</code></a></li>
<li>3.3 <a href="#s3.3">Container elements</a>
<ul class="tocline">
<li>3.3.1 <a href="#s3.3.1"><code>emma:one-of</code>
element</a></li>
<li>3.3.2 <a href="#s3.3.2"><code>emma:group</code> element</a>
<ul class="tocline">
<li>3.3.2.1 <a href="#s3.3.2.1">Indirect grouping criteria:
<code>emma:group-info</code> element</a></li>
</ul>
</li>
<li>3.3.3 <a href="#s3.3.3"><code>emma:sequence</code>
element</a></li>
</ul>
</li>
<li>3.4 <a href="#s3.4">Lattice element</a>
<ul class="tocline">
<li>3.4.1 <a href="#s3.4.1">Lattice markup:
<code>emma:lattice</code>, <code>emma:arc</code>,
<code>emma:node</code> elements</a></li>
<li>3.4.2 <a href="#s3.4.2">Annotations on lattices</a></li>
<li>3.4.3 <a href="#s3.4.3">Relative timestamps on
lattices</a></li>
</ul>
</li>
<li>3.5 <a href="#s3.5">Literal semantics:
<code>emma:literal</code> element</a></li>
</ul>
</li>
<li>4 <a href="#s4">EMMA annotations</a>
<ul class="tocline">
<li>4.1 <a href="#s4.1">EMMA annotation elements</a>
<ul class="tocline">
<li>4.1.1 <a href="#s4.1.1">Data model: <code>emma:model</code>
element</a></li>
<li>4.1.2 <a href="#s4.1.2">Interpretation derivation:
<code>emma:derived-from</code> element and
<code>emma:derivation</code> element</a></li>
<li>4.1.3 <a href="#s4.1.3">Reference to grammar used:
<code>emma:grammar</code> element</a></li>
<li>4.1.4 <a href="#s4.1.4">Extensibility to application/vendor
specific annotations: <code>emma:info</code> element</a></li>
<li>4.1.5 <a href="#s4.1.5">Endpoint reference:
<code>emma:endpoint-info</code> element and
<code>emma:endpoint</code> element</a></li>
</ul>
</li>
<li>4.2 <a href="#s4.2">EMMA annotation attributes</a>
<ul class="tocline">
<li>4.2.1 <a href="#s4.2.1">Tokens of input:
<code>emma:tokens</code> attribute</a></li>
<li>4.2.2 <a href="#s4.2.2">Reference to processing:
<code>emma:process</code> attribute</a></li>
<li>4.2.3 <a href="#s4.2.3">Lack of input:
<code>emma:no-input</code> attribute</a></li>
<li>4.2.4 <a href="#s4.2.4">Uninterpreted input:
<code>emma:uninterpreted</code> attribute</a></li>
<li>4.2.5 <a href="#s4.2.5">Human language of input:
<code>emma:lang</code> attribute</a></li>
<li>4.2.6 <a href="#s4.2.6">Reference to signal:
<code>emma:signal</code> <span>and
<code>emma:signal-size</code></span> attributes</a></li>
<li>4.2.7 <a href="#s4.2.7">Media type:
<code>emma:media-type</code> attribute</a></li>
<li>4.2.8 <a href="#s4.2.8">Confidence scores:
<code>emma:confidence</code> attribute</a></li>
<li>4.2.9 <a href="#s4.2.9">Input source: <code>emma:source</code>
attribute</a></li>
<li>4.2.10 <a href="#s4.2.10">Timestamps</a>
<ul class="tocline">
<li>4.2.10.1 <a href="#s4.2.10.1">Absolute timestamps:
<code>emma:start</code>, <code>emma:end</code> attributes</a></li>
<li>4.2.10.2 <a href="#s4.2.10.2">Relative timestamps:
<code>emma:time-ref-uri</code>,
<code>emma:time-ref-anchor-point</code>,
<code>emma:offset-to-start</code> attributes</a></li>
<li>4.2.10.3 <a href="#s4.2.10.3">Duration of input:
<code>emma:duration</code> attribute</a></li>
<li><span>4.2.10.4 <a href="#s4.2.10.4">Composite Input and
Relative Timestamps</a></span></li>
</ul>
</li>
<li>4.2.11 <a href="#s4.2.11">Medium, mode, and function of user
inputs: <code>emma:medium</code>, <code>emma:mode</code>,
<code>emma:function</code>, <code>emma:verbal</code>
attributes</a></li>
<li>4.2.12 <a href="#s4.2.12">Composite multimodality:
<code>emma:hook</code> attribute</a></li>
<li>4.2.13 <a href="#s4.2.13">Cost: <code>emma:cost</code>
attribute</a></li>
<li>4.2.14 <a href="#s4.2.14">Endpoint properties:
<code>emma:endpoint-role</code>,
<code>emma:endpoint-address</code>, <code>emma:port-type</code>,
<code>emma:port-num</code>, <code>emma:message-id</code>,
<code>emma:service-name</code>, <code>emma:endpoint-pair-ref</code>,
<code>emma:endpoint-info-ref</code>
attributes</a></li>
<li>4.2.15 <a href="#s4.2.15">Reference to
<code>emma:grammar</code> element: <code>emma:grammar-ref</code>
attribute</a></li>
<li>4.2.16 <a href="#s4.2.16">Reference to <code>emma:model</code>
element: <code>emma:model-ref</code> attribute</a></li>
<li>4.2.17 <a href="#s4.2.17">Dialog turns:
<code>emma:dialog-turn</code> attribute</a></li>
</ul>
</li>
<li>4.3 <a href="#s4.3">Scope of EMMA annotations</a></li>
</ul>
</li>
<li>5.<a href="#s5">Conformance</a>
<ul class="tocline">
<li>5.1 <a href="#s5.1">Conforming EMMA Documents</a></li>
<li>5.2 <a href="#s5.2">Using EMMA with other Namespaces</a></li>
<li>5.3 <a href="#s5.3">Conforming EMMA Processors</a></li>
</ul>
</li>
<li><a href="#appendices">Appendices</a>
<ul class="tocline">
<li>Appendix A. <a href="#appA">XML and <span>RELAX NG</span>
schemata</a> <span>(Normative)</span></li>
<li>Appendix B. <a href="#appB">MIME type</a>
<span>(Normative)</span>
<ul>
<li><span>B.1 <a href="#media-type-registration">Registration of
MIME media type application/emma+xml</a></span></li>
</ul>
</li>
<li>Appendix C. <a href="#appC"><code>emma:hook</code> and SRGS</a>
<span>(Informative)</span></li>
<li>Appendix D. <a href="#appD">EMMA event interface</a>
<span>(Informative)</span></li>
<li>Appendix E. <a href="#appE">References</a>
<ul>
<li>E.1 <a href="#appE1">Normative references</a></li>
<li>E.2 <a href="#appE2"><span>Informative</span>
references</a></li>
</ul>
</li>
<li>Appendix F. <a href="#appF">Changes since last draft</a>
<span>(Informative)</span></li>
<li>Appendix G. <a href="#appG">Acknowledgements</a>
<span>(Informative)</span></li>
</ul>
</li>
</ul>
<h2 id="s1">1. Introduction</h2>
<p>This section is <span>I</span>nformative.</p>
<p>This document presents an XML specification for EMMA, an
Extensible MultiModal Annotation markup language, responding to the
requirements documented in <span>Requirements for EMMA</span>
[<a href="#EMMAreqs">EMMA <span>Requirements</span></a>]. This
markup language is intended for use by systems that provide
semantic interpretations for a variety of inputs, including but not
necessarily limited to, speech, natural language text, GUI and ink
input.</p>
<p>It is expected that this markup will be used primarily as a
standard data interchange format between the components of a
multimodal system; in particular, it will normally be automatically
generated by interpretation components to represent the semantics
of users' inputs, not directly authored by developers.</p>
<p>The language is focused on annotating single inputs from users,
which may be either from a single mode or a composite input
combining information from multiple modes, as opposed to
information that might have been collected over multiple turns of a
dialog. The language provides a set of elements and attributes that
are focused on enabling annotations on user inputs and
interpretations of those inputs.</p>
<p>An EMMA document can be considered to hold three types of
data:</p>
<ul>
<li>
<p><b>instance data</b></p>
<p>Application-specific markup corresponding to input information
which is meaningful to the consumer of an EMMA document. Instances
are application-specific and built by input processors at runtime.
Given that utterances may be ambiguous with respect to input
values, an EMMA document may hold more than one instance.</p>
</li>
<li>
<p><b>data model</b></p>
<p>Constraints on structure and content of an instance. The data
model is typically pre-established by an application, and may be
implicit, that is, unspecified.</p>
</li>
<li>
<p><b>metadata</b></p>
<p>Annotations associated with the data contained in the instance.
Annotation values are added by input processors at runtime.</p>
</li>
</ul>
<p>Given the assumptions above about the nature of data represented
in an EMMA document, the following general principles apply to the
design of EMMA:</p>
<ul>
<li>The main prescriptive content of the EMMA specification will
consist of metadata: EMMA will provide a means to express the
metadata annotations which require standardization. (Notice,
however, that such annotations may express the relationship among
all the types of data within an EMMA document.)</li>
<li>The instance and its data model are assumed to be specified in
XML, but EMMA will remain agnostic to the XML format used to
express these. (The instance XML is assumed to be sufficiently
structured to enable the association of annotative data.)</li>
<li>The extensibility of EMMA lies in the ability for additional
kinds of metadata to be included in application specific
vocabularies. EMMA itself can be extended with application and
vendor specific annotations contained within the
<code>emma:info</code> element <span>(<a href="#s4.1.4">Section
4.1.4</a>)</span>.</li>
</ul>
<p>The annotations of EMMA should be considered 'normative' in the
sense that if an EMMA component produces annotations as described
in <a href="#s3">Section 3</a> <span>and <a href="#s4">Section
4</a></span>, these annotations must be represented using the EMMA
syntax. The Multimodal Interaction Working Group may address in
later drafts the issues of modularization and profiling; that is,
which sets of annotations are to be supported by which classes of
EMMA component.</p>
<h3 id="s1.1">1.1 Uses of EMMA</h3>
<p>The general purpose of EMMA is to represent information
automatically extracted from a user's input by an interpretation
component, where input is to be taken in the general sense of a
meaningful user input in any modality supported by the platform.
The reader should refer to the sample architecture in <span>W3C
Multimodal Interaction Framework</span> <a href="#MMIF">[<span>MMI
Framework</span>]</a>, which shows EMMA conveying content between
user input modality components and an interaction manager.</p>
<p>Components that generate EMMA markup:</p>
<ol>
<li>Speech recognizers</li>
<li>Handwriting recognizers</li>
<li>Natural language understanding engines</li>
<li>Other input media interpreters (e.g. DTMF, pointing,
keyboard)</li>
<li>Multimodal integration component</li>
</ol>
<p>Components that use EMMA include:</p>
<ol>
<li>Interaction manager</li>
<li>Multimodal integration component</li>
</ol>
<p>Although not a primary goal of EMMA, a platform may also choose
to use this general format as the basis of a general semantic
result that is carried along and filled out during each stage of
processing. In addition, future systems may also potentially make
use of this markup to convey abstract semantic content to be
rendered into natural language by a natural language generation
component.</p>
<h3 id="s1.2">1.2 Terminology</h3>
<dl>
<dt id="anchor-point">anchor point</dt>
<dd>When referencing an input interval with
<code>emma:time-ref-uri</code>,
<code>emma:time-ref-anchor-point</code> allows you to specify
whether the referenced anchor is the start or end of the
interval.</dd>
<dt id="annotation">annotation</dt>
<dd>Information about the interpreted input, for example,
timestamps, confidence scores, links to raw input, etc.</dd>
<dt id="composite-input">composite input</dt>
<dd>An input formed from several pieces, often in different modes,
for example, a combination of speech and pen gesture, such as
saying "zoom in here" and circling a region on a map.</dd>
<dt id="confidence">confidence</dt>
<dd>A numerical score describing the degree of certainty in a
particular interpretation of user input.</dd>
<dt id="data-model">data model</dt>
<dd>For EMMA, a data model defines a set of constraints on possible
interpretations of user input.</dd>
<dt id="derivation">derivation</dt>
<dd>Interpretations of user input are said to be derived from that
input, and higher level interpretations may be derived from lower
level ones. EMMA allows you to reference the user input or
interpretation a given interpretation was derived from, see
<a href="#semantic-interpretation"><em>semantic
interpretation</em></a>.</dd>
<dt id="dialog">dialog</dt>
<dd>For EMMA, dialog can be considered as a sequence of
interactions between 
a user and the application.</dd>
<dt id="endpoint">endpoint</dt>
<dd>In EMMA, this refers to a network location which is the source
or recipient of an EMMA document. It should be noted that the usage
of the term "endpoint" in this context is different from the way
that the term is used in speech processing, where it refers to the
end of a speech input.</dd>
<dt id="gestures">gestures</dt>
<dd>In multimodal applications gestures are communicative acts made
by the user or application. An example is circling an area on a map
to indicate a region of interest. Users may be able to gesture with
a pen, keystrokes, hand movements, head
movements, or sound. Gestures often form part of <a href=
"#composite-input"><em>composite input</em></a>. Application
gestures are typically animations and/or sound effects.</dd>
<dt id="grammar">grammar</dt>
<dd>A set of rules that describe a sequence of tokens expected in a
given input. These can be used by speech and handwriting
recognizers to increase recognition accuracy.</dd>
<dt id="handwriting-recognition">handwriting recognition</dt>
<dd>The process of converting pen strokes into text.</dd>
<dt id="ink-recognition">ink recognition</dt>
<dd>This includes the recognition of handwriting and pen
gestures.</dd>
<dt id="input-cost">input cost</dt>
<dd>In EMMA, this refers to a numerical measure indicating the
weight or processing cost associated with a user's input or part of
their input.</dd>
<dt id="input-device">input device</dt>
<dd>The device proving a particular input, for example, a
microphone, a pen, a mouse, a camera, or a keyboard.</dd>
<dt id="input-function">input function</dt>
<dd>In EMMA, this refers to <span>the</span> use a particular input
is serving, for example, as part of a recording or transcription,
as part of a dialog, or as a means to verify the user's
identity.</dd>
<dt id="input-medium">input medium</dt>
<dd>Whether the input is acoustic, visual, or tactile, for
instance, a spoken utterance is an example of an aural input, a
hand gesture as seen by a camera is an example of a visual input,
pointing with a mouse or pen is an example of a tactile input.</dd>
<dt id="input-mode">input mode</dt>
<dd>This distinguishes a particular means of providing an input
within a general input medium, for example, speech, DTMF, ink, key
strokes, video, photograph, etc.</dd>
<dt id="input-source">input source</dt>
<dd>This is the device that provided the input, for example a
particular microphone or camera. EMMA allows you to identify these
with a URI.</dd>
<dt id="input-tokens">input tokens</dt>
<dd>In EMMA, this refers to a sequence of characters, words or
other discrete units of input.</dd>
<dt id="instance-data">instance data</dt>
<dd>A representation in XML of an interpretation of user
input.</dd>
<dt id="interaction-manager">interaction manager</dt>
<dd>A processor that determines how an application interacts with a
user. This can be at multiple levels of abstraction, for example,
at a detailed level, determining what prompts to present to the
user and what actions to take in response to user input, versus a
higher level treatment in terms of goals and tasks for achieving
those goals. Interaction managers are frequently event driven.</dd>
<dt id="interpretation">interpretation</dt>
<dd>In EMMA, an interpretation of user input refers to information
derived from the user input that is meaningful to the
application.</dd>
<dt id="keystroke-input">keystroke input</dt>
<dd>Input provided by the user pressing on a sequence of keys
(buttons), such as a computer keyboard or keypad.</dd>
<dt id="lattice">lattice</dt>
<dd>A set of nodes interconnected with directed arcs such that by
following an arc, you can never find yourself back at a node you
have already visited (i.e. a directed acyclic graph). Lattices
provide a flexible means to represent the results of speech and
handwriting recognition, in terms of arcs representing words or
character sequences. Different arcs from the same node represent
different local hypotheses as to what the user said or wrote.</dd>
<dt id="metadata">metadata</dt>
<dd>Information describing another set of data, for instance, a
library catalog card with information on the author, title and
location of a book. EMMA is designed to support input processors in
providing metadata for interpretations of user input.</dd>
<dt id="multimodal-integration">multimodal integration</dt>
<dd>The process of combining inputs from different modes to create
an interpretation of composite input. This is also sometimes
referred to as <em>multimodal fusion</em>.</dd>
<dt id="multimodal-interaction">multimodal interaction</dt>
<dd>The means for a user to interact with an application using more
than one mode of interaction, for instance, offering the user the
choice of speaking or typing, or in some cases, allowing the user
to provide a composite input involving multiple modes.</dd>
<dt id="natural-language-understanding">natural language
understanding</dt>
<dd>The process of interpreting text in terms that are useful for
an application.</dd>
<dt id="N-best-list">N-best list</dt>
<dd>An N-best list is a list of the most likely hypotheses for what
the user actually said or wrote, where N stands for an integral
number such as 5 for the 5 most likely hypotheses.</dd>
<dt id="raw-signal">raw signal</dt>
<dd>An uninterpreted input, such as an audio waveform captured from
a microphone.</dd>
<dt id="semantic-interpretation">semantic interpretation</dt>
<dd>A normalized representation of the meaning of a user input, for
instance, mapping the speech for "San Francisco" into the airport
code "SFO".</dd>
<dt id="semantic-processor">semantic processor</dt>
<dd>In EMMA, this refers to systems that can derive interpretations
of user input, for instance, mapping the speech for "San Francisco"
into the airport code "SFO".</dd>
<dt id="signal-interpretation">signal interpretation</dt>
<dd>The process of mapping a discrete or continuous signal into a
symbolic representation that can be used by an application, for
instance, transforming the audio waveform corresponding to someone
saying "2005" into the number 2005.</dd>
<dt id="speech-recognition">speech recognition</dt>
<dd>The process of determining the textual transcription of a piece
of speech.</dd>
<dt id="speech-synthesis">speech synthesis</dt>
<dd>The process of rendering a piece of text into the corresponding
speech, i.e. synthesi<span>z</span>ing speech from text.</dd>
<dt id="text-to-speech">text to speech</dt>
<dd>The process of rendering a piece of text into the corresponding
speech.</dd>
<dt id="time-stamp">time stamp</dt>
<dd>The time that a particular input or part of an input began or
ended.</dd>
<dt id="term-uri">URI: Uniform Resource Identifier</dt>
<dd>A URI is a unifying syntax for the expression of names and
addresses of objects on the network as used in the World Wide Web.
<span>Within this specification, the term URI refers to a Universal
Resource Identifier as defined in [<a href="#RFC3986">RFC3986</a>]
and extended in [<a href="#RFC3987">RFC3987</a>] with the new name
IRI. The term URI has been retained in preference to IRI to avoid
introducing new names for concepts such as "Base URI" that are
defined or referenced across the whole family of XML
specifications</span>. A URI is defined as any legal
<code>anyURI</code> primitive as defined in XML Schema Part 2:
Datatypes Second Edition Section 3.2.17 [<a href=
"#XSD2">SCHEMA2</a>].</dd>
<dt id="user-input">user input</dt>
<dd>An input provided by a user as opposed to something generated
automatically.</dd>
</dl>
<h2 id="s2">2. Structure of EMMA documents</h2>
<p>This section is <span>I</span>nformative.</p>
<p>As noted above, the main components of an interpreted user input
in EMMA are the instance data, an optional data model, and the
metadata annotations that may be applied to that input. The
realization of these components in EMMA is as follows:</p>
<ul>
<li><b>instance data</b> is contained within an EMMA
<i>interpretation</i></li>
<li>the <b>data model</b> is optionally specified as an annotation
of that instance</li>
<li>EMMA <b>annotations</b> may be applied at different levels of
an EMMA document.</li>
</ul>
<p>An EMMA <i>interpretation</i> is the primary unit for holding
user input as interpreted by an EMMA processor. As will be seen
below, multiple interpretations of a single input are possible.</p>
<p>EMMA provides a simple structural syntax for the organization of
interpretations and instances, and an annotative syntax to apply
the annotation to the input data at different levels.</p>
<p>An outline of the structural syntax and annotations found in
EMMA documents is as follows. A fuller definition may be found in
the description of individual elements and attributes in <a href=
"#s3"><span>S</span>ection 3</a> and <a href=
"#s4"><span>S</span>ection 4</a>.</p>
<ul>
<li><b><a href="#s3">EMMA <span>s</span>tructural
<span>e</span>lements</a></b> (<a href="#s3">Section 3</a>)
<ul>
<li><b><a href="#s3.1">Root element</a></b>: The root node of an
EMMA document, the <code>emma:emma</code> element, holds EMMA
version and namespace information, and provides a container for one
or more of the following interpretation and container elements
(<a href="#s3.1">Section 3.1</a>)</li>
<li><b><a href="#s3.2">Interpretation element</a></b>: The
<code>emma:interpretation</code> element contains a given
interpretation of the input and holds application specific markup
(<a href="#s3.2">Section 3.2</a>)</li>
<li><b><a href="#s3.3">Container elements</a>:</b>
<ul>
<li><code>emma:one-of</code> is a container for one or more
interpretation elements or container elements and denotes that
these are mutually exclusive interpretations (<a href=
"#s3.3.1">Section 3.3.1</a>)</li>
<li><code>emma:group</code> is a general container for one or more
interpretation elements or container elements. It can be associated
with arbitrary grouping criteria (<a href="#s3.3.2">Section
3.3.2</a>).</li>
<li><code>emma:sequence</code> is a container for one or more
interpretation elements or container elements and denotes that
these are sequential in time (<a href="#s3.3.3">Section
3.3.3</a>).</li>
</ul>
</li>
<li><b><a href="#s3.4">Lattice element</a></b>: The
<code>emma:lattice</code> element is used to contain a series of
<code>emma:arc</code> and <code>emma:node</code> elements that
define a lattice of words, gestures, meanings or other symbols. The
<code>emma:lattice</code> element appears within the
<code>emma:interpretation</code> element (<a href="#s3.4">Section
3.4</a>)</li>
<li><b><a href="#s3.5">Literal element</a></b>: The
<code>emma:literal</code> element is used as a wrapper when the
application semantics is a string literal. (<a href="#s3.5">Section
3.5</a>)</li>
</ul>
</li>
<li><b><a href="#s4">EMMA annotations</a></b> (<a href=
"#s4">Section 4</a>)
<ul>
<li><b><a href="#s4.1">EMMA annotation elements</a></b>: These are
EMMA annotations such as <code>emma:derived-from</code>,
<code>emma:endpoint-info</code>, and <code>emma:info</code> which
are represented as elements so that they can occur more than once
within an element and can contain internal structure. (<a href=
"#s4.1">Section 4.1</a>)</li>
<li><b><a href="#s4.2">EMMA annotation attributes</a></b>: These
are EMMA annotations such as <code>emma:start</code>,
<code>emma:end</code> , <code>emma:confidence</code>, and
<code>emma:tokens</code> which are represented as attributes. They
can appear on <code>emma:interpretation</code> elements<span>.
S</span>ome can appear on container elements, lattice elements, and
elements in the application-specific markup. (<a href=
"#s4.2">Section 4.2</a>)</li>
</ul>
</li>
</ul>
<p>From the defined root node <code>emma:emma</code> the structure
of an EMMA document consists of a tree of EMMA container elements
(<code>emma:one-of</code>, <code>emma:sequence</code>,
<code>emma:group</code>) terminating in a number of interpretation
elements (<code>emma:interpretation</code>). The
<code>emma:interpretation</code> elements serve as wrappers for
either application namespace markup describing the interpretation
of the users input or an <code>emma:lattice</code> element or
<code>emma:literal</code> element . A single
<code>emma:interpretation</code> may also appear directly under the
root node.</p>


<p>
The EMMA elements
<code>emma:emma</code>,
<code>emma:interpretation</code>,
<code>emma:one-of</code>,
and <code>emma:literal</code>
and the EMMA attributes
<code>emma:no-input</code>,
<code>emma:uninterpreted</code>,
<code>emma:medium</code>,
and <code>emma:mode</code>
are required of all
implementations.  The remaining elements and attributes are optional
and may be used in some implementations and not other depending on the
specific modalities and processing being represented.
</p>


<p>To illustrate this, here is an example <span class="new">of
an</span> EMMA document <span class="new">representing</span> input
to a flight reservation application. In this example there are two
speech recognition results and associated semantic representations
of the input. The system is uncertain whether the user meant
"flights from Boston to Denver" or "flights from Austin to Denver".
The annotations to be captured are timestamps and confidence scores
for the two inputs.</p>
<p>Example:</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:one-of id="r1" emma:start="1087995961542" emma:end="1087995963542"
<span>     emma:medium="acoustic" emma:mode="voice"</span>&gt;
    &lt;emma:interpretation id="int1" emma:confidence="0.75"
    emma:tokens="flights from boston to denver"&gt;
      &lt;origin&gt;Boston&lt;/origin&gt;
      &lt;destination&gt;Denver&lt;/destination&gt;
    &lt;/emma:interpretation&gt;

    &lt;emma:interpretation id="int2" emma:confidence="0.68"
    emma:tokens="flights from austin to denver"&gt;
      &lt;origin&gt;Austin&lt;/origin&gt;
      &lt;destination&gt;Denver&lt;/destination&gt;
    &lt;/emma:interpretation&gt;
  &lt;/emma:one-of&gt;
&lt;/emma:emma&gt;
</pre>
<p>Attributes on the root <code>emma:emma</code> element indicate
the version and namespace. The <code>emma:emma</code> element
contains an <code>emma:one-of</code> element which contains a
disjunctive list of possible interpretations of the input. The
actual semantic representation of each interpretation is within the
application namespace. In the example here the application specific
semantics involves elements <code>origin</code> and
<code>destination</code> indicating the origin and destination
cities for looking up a flight. The timestamp is the same for both
interpretations and it is annotated using values in milliseconds in
the <code>emma:start</code> and <code>emma:end</code> attributes on
the <code>emma:one-of</code>. The confidence scores and tokens
associated with each of the inputs are annotated using the EMMA
annotation attributes <code>emma:confidence</code> and
<code>emma:tokens</code> on each of the
<code>emma:interpretation</code> elements.</p>
<h3 id="s2.1">2.<span>1</span> Data model</h3>
<p>An EMMA data model expresses the constraints on the structure
and content of instance data, for the purposes of validation. As
such, the data model may be considered as a particular kind of
annotation (although, unlike other EMMA annotations, it is not a
feature pertaining <span>to</span> a specific user input at a
specific moment in time, it is rather a static and, by its very
definition, application-specific structure). <span>The</span>
specification of <span>a data model</span> in EMMA is optional.</p>
<p>Since Web applications today use different formats to specify
data models, e.g. <span>XML Schema Part 1: Structures Second
Edition</span> [<a href="#XSD1">XML Schema
<span>Structures</span></a>], XForms <span>1.0 (Second
Edition)</span> [<a href="#XFORMS">XFORMS</a>], <span>RELAX NG
Specification</span> [<a href="#RELAXNG">RELAX-NG</a>], etc., EMMA
itself is agnostic to the format of data model used.</p>
<p>Data model definition and reference is defined in <a href=
"#s4.1.1">Section 4.1.1</a>.</p>
<h3 id="s2.2">2.<span>2</span> EMMA namespace prefixes</h3>
<p>An EMMA attribute is qualified with the EMMA namespace prefix if
the attribute can also be used as an in-line annotation on elements
in the application's namespace. Most of the EMMA annotation
attributes in <a href="#s4.2">Section 4.2</a> are in this category.
An EMMA attribute is not qualified with the EMMA namespace prefix
if the attribute only appears on an EMMA element. This rule ensures
consistent usage of the attributes across all examples.</p>
<p>Attributes from other namespaces are permissible on all EMMA
elements. As an example <code>xml:lang</code> may be used to
annotate the human language of character data content.</p>
<h2 id="s3">3. EMMA structural elements</h2>
<p>This section defines elements in the EMMA namespace which
provide the structural syntax of EMMA documents.</p>
<h3 id="s3.1">3.1 Root element: <code>emma:emma</code></h3>
<table class="defn" summary="property definition" width="98%"
cellpadding="5" cellspacing="0">
<tbody>
<tr>
<th>Annotation</th>
<th>emma:emma</th>
</tr>
<tr>
<th>Definition</th>
<td>The root element of an EMMA document.</td>
</tr>
<tr>
<th>Children</th>
<td>The <code>emma:emma</code> element MUST immediately contain a
single <code>emma:interpretation</code> element or EMMA container
element: <code>emma:one-of</code>, <code>emma:group</code>,
<code>emma:sequence</code>. It MAY also contain an optional single
<code>emma:derivation</code> element and an optional single
<code>emma:info</code> annotation element. It MAY also contain
multiple optional <code>emma:grammar</code> annotation elements,
<code>emma:model</code> annotation elements, and
<code>emma:endpoint-info</code> annotation elements.</td>
</tr>
<tr>
<th>Attributes</th>
<td>
<ul>
<li><b>Required</b>:
<ul>
<li><code>version</code>: the version of EMMA used for the
interpretation(s). Interpretations expressed using this
specification MUST use <code>1.0</code> for the value.</li>
<li>Namespace declaration for EMMA, see below.</li>
</ul>
</li>
<li><b>Optional</b>:
<ul>
<li>any other namespace declarations for application specific
namespaces.</li>
</ul>
</li>
</ul>
</td>
</tr>
<tr>
<th>Applies to</th>
<td>None</td>
</tr>
</tbody>
</table>
<p>The root element of an EMMA document is named
<code>emma:emma</code>. It holds a single
<code>emma:interpretation</code> or EMMA container element
(<code>emma:one-of</code>, <code>emma:sequence</code>,
<code>emma:group</code>). It MAY also contain a single
<code>emma:derivation</code> element containing earlier stages of
the processing of the input (See <a href="#s4.1.2">Section
4.1.2</a>). It MAY also contain an optional single annotation
element: <code>emma:info</code> and multiple optional
<code>emma:grammar</code>, <code>emma:model</code>, and
<code>emma:endpoint-info</code> elements.</p>
<p>It MAY hold attributes for information pertaining to EMMA
itself, along with any namespaces which are declared for the entire
document, and any other EMMA annotative data. The
<code>emma:emma</code> element and other elements and attributes
defined in this specification belong to the XML namespace
identified by the URI "http://www.w3.org/2003/04/emma". In the
examples, the EMMA namespace is generally declared using the
attribute <code>xmlns:emma</code> on the root
<code>emma:emma</code> element. EMMA processors MUST support the
full range of ways of declaring XML namespaces as defined by the
<span>Namespaces in XML 1.1 (Second Edition)</span> [<a href=
"#XMLNS">XMLNS</a>]. Application markup MAY be declared in an
explicit application namespace, or an undefined namespace
(equivalent to setting xmlns="").</p>
<p>For example:</p>
<pre class="example">
&lt;emma:emma version="1.0" xmlns:emma="http://www.w3.org/2003/04/emma"&gt;
    ....
&lt;/emma:emma&gt;
</pre>
<p>or</p>
<pre class="example">
&lt;emma version="1.0" xmlns="http://www.w3.org/2003/04/emma"&gt;
    ....
&lt;/emma&gt;
</pre>
<h3 id="s3.2">3.2 Interpretation element:
<code>emma:interpretation</code></h3>
<table class="defn" summary="property definition" width="98%"
cellpadding="5" cellspacing="0">
<tbody>
<tr>
<th>Annotation</th>
<th>emma:interpretation</th>
</tr>
<tr>
<th>Definition</th>
<td>The <code>emma:interpretation</code> element acts as a wrapper
for application instance data or lattices.</td>
</tr>
<tr>
<th>Children</th>
<td>The <code>emma:interpretation</code> element MUST immediately
contain either application instance data, or a single
<code>emma:lattice</code> element, or a single
<code>emma:literal</code> element, or in the case of uninterpreted
input or no input <code>emma:interpretation</code>
<span>MUST</span> be empty. It MAY also contain <span>multiple
optional</span> <code>emma:derived-from</code>
element<span>s</span> and <span>an optional single</span>
<code>emma:info</code> <span>element</span>.</td>
</tr>
<tr>
<th>Attributes</th>
<td>
<ul>
<li><b>Required</b>: Attribute <code>id</code> of type
<code>xsd:ID</code> that uniquely identifies the interpretation
within the EMMA document.</li>
<li><b>Optional</b>: The annotation attributes:
<code>emma:tokens</code>, <code>emma:process</code>,
<code>emma:no-input</code>, <code>emma:uninterpreted</code>,
<code>emma:lang</code>, <code>emma:signal</code>,
<code><span>emma:signal-size</span></code>,
<code>emma:media-type</code>, <code>emma:confidence</code>,
<code>emma:source</code>, <code>emma:start</code>,
<code>emma:end</code>, <code>emma:time-ref-uri</code>,
<code>emma:time-ref-anchor-point</code>,
<code>emma:offset-to-start</code>, <code>emma:duration</code>,
<code>emma:medium</code>, <code>emma:mode</code>,
<code>emma:function</code>, <code>emma:verbal</code>,
<code>emma:cost</code>, <code>emma:grammar-ref</code>,
<code>emma:endpoint-info-ref</code>, <code>emma:model-ref</code>,
<code>emma:dialog-turn</code>.</li>
</ul>
</td>
</tr>
<tr>
<th>Applies to</th>
<td>The <code>emma:interpretation</code> element is legal only as a
child of <code>emma:emma</code>, <code>emma:group</code>,
<code>emma:one-of</code>, <code>emma:sequence</code>, or
<code>emma:derivation</code>.</td>
</tr>
</tbody>
</table>
<p>The <code>emma:interpretation</code> element holds a single
interpretation represented in application specific markup, or a
single <code>emma:lattice</code> element, or a single
<code>emma:literal</code> element.</p>
<p>The <code>emma:interpretation</code> element MUST be empty if it
is marked with <code>emma:no-input="true"</code> <span>(<a href=
"#s4.2.3">Section 4.2.3</a>)</span>. The
<code>emma:interpretation</code> element <span>MUST</span> be empty
if it has been annotated with
<code>emma:uninterpreted="true"</code> <span>(<a href=
"#s4.2.4">Section 4.2.4</a>)</span> or
<code>emma:function="recording"</code> <span>(<a href=
"#s4.2.11">Section 4.2.11</a>)</span>.</p>
<p>Attributes:</p>
<ol>
<li><b>id</b> a REQUIRED <code>xsd:ID</code> value that uniquely
identifies the interpretation within the EMMA document.</li>
</ol>
<pre class="example">
&lt;emma:emma version="1.0" xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:interpretation id="r1" emma:medium="acoustic" emma:mode="voice"&gt;
    ...
  &lt;/emma:interpretation&gt;
&lt;/emma:emma&gt;
</pre>
<p>While <code>emma:medium</code> and <code>emma:mode</code> are
optional on <code>emma:interpretation</code>, note that all EMMA
interpretations must be annotated for <code>emma:medium</code> and
<code>emma:mode</code>, so either these attributes must appear
directly on <code>emma:interpretation</code> or they must appear on
an ancestor <code>emma:one-of</code> node or they must appear on an
earlier stage of the derivation listed in
<code>emma:derivation</code>.</p>
<h3 id="s3.3">3.3 Container elements</h3>
<h3 id="s3.3.1">3.3.1 <code>emma:one-of</code> element</h3>
<table class="defn" summary="property definition" width="98%"
cellpadding="5" cellspacing="0">
<tbody>
<tr>
<th>Annotation</th>
<th>emma:one-of</th>
</tr>
<tr>
<th>Definition</th>
<td>A container element indicating a disjunction among a collection
of mutually exclusive interpretations of the input.</td>
</tr>
<tr>
<th>Children</th>
<td>The <code>emma:one-of</code> element MUST immediately contain a
collection of one or more <code>emma:interpretation</code> elements
or container elements: <code>emma:one-of</code>,
<code>emma:group</code>, <code>emma:sequence</code> . It MAY also
contain <span>multiple optional</span>
<code>emma:derived-from</code> element<span>s</span> and <span>an
optional single</span> <code>emma:info</code>
<span>element</span>.</td>
</tr>
<tr>
<th>Attributes</th>
<td>
<ul>
<li><b>Required</b>:
<ul>
<li>Attribute <code>id</code> of type <code>xsd:ID</code></li>
<li>The attribute <code>disjunction-type</code> MUST be present if
<code>emma:one-of</code> is embedded within
<code>emma:one-of</code>. <span>The possible values of
<code>disjunction-type</code> are {<code>recognition</code>,
<code>understanding</code>, <code>multi-device</code>, and
<code>multi-process</code>}.</span></li>
</ul>
</li>
<li><b>Optional</b>:
<ul>
<li>On a single non-embedded <code>emma:one-of</code> the attribute
<code>disjunction-type</code> is optional.</li>
<li>The following annotation attributes are optional:
<code>emma:tokens</code>, <code>emma:process</code>,
<code>emma:lang</code>, <code>emma:signal</code>,
<code><span>emma:signal-size</span></code>,
<code>emma:media-type</code>, <code>emma:confidence</code>,
<code>emma:source</code>, <code>emma:start</code>,
<code>emma:end</code>, <code>emma:time-ref-uri</code>,
<code>emma:time-ref-anchor-point</code>,
<code>emma:offset-to-start</code>, <code>emma:duration</code>,
<code>emma:medium</code>, <code>emma:mode</code>,
<code>emma:function</code>, <code>emma:verbal</code>,
<code>emma:cost</code>, <code>emma:grammar-ref</code>,
<code>emma:endpoint-info-ref</code>, <code>emma:model-ref</code>,
<code>emma:dialog-turn</code>.</li>
</ul>
</li>
</ul>
</td>
</tr>
<tr>
<th>Applies to</th>
<td>The <code>emma:one-of</code> element MAY only appear as a child
of <code>emma:emma</code>, <code>emma:one-of</code>,
<code>emma:group</code>, <code>emma:sequence</code>, or
<code>emma:derivation</code>.</td>
</tr>
</tbody>
</table>
<p>The <code>emma:one-of</code> element acts as a container for a
collection of one or more interpretation
(<code>emma:interpretation</code>) or container elements
(<code>emma:one-of</code>, <code>emma:group</code>,
<code>emma:sequence</code>), and denotes that these are mutually
exclusive interpretations.</p>
<p>An N-best list of choices in EMMA MUST be represented as a set
of <code>emma:interpretation</code> elements contained within an
<code>emma:one-of</code> element. For instance, a series of
different recognition results in speech recognition might be
represented in this way.</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:one-of id="r1" <span>emma:medium="acoustic" emma:mode="voice"</span>&gt;
    &lt;emma:interpretation id="int1"&gt;
      &lt;origin&gt;Boston&lt;/origin&gt;
      &lt;destination&gt;Denver&lt;/destination&gt;
      &lt;date&gt;03112003&lt;/date&gt;
    &lt;/emma:interpretation&gt;

    &lt;emma:interpretation id="int2"&gt;
      &lt;origin&gt;Austin&lt;/origin&gt;
      &lt;destination&gt;Denver&lt;/destination&gt;
      &lt;date&gt;03112003&lt;/date&gt;
    &lt;/emma:interpretation&gt;
  &lt;/emma:one-of&gt;
&lt;/emma:emma&gt;
</pre>
<p>The function of the <code>emma:one-of</code> element is to
represent a disjunctive list of possible interpretations of a user
input. A disjunction of possible interpretations of an input can be
the result of different kinds of processing or ambiguity. One
source is multiple results from a recognition technology such as
speech or handwriting recognition. Multiple results can also occur
from parsing or understanding natural language. Another possible
source of ambiguity is from the application of multiple different
kinds of recognition or understanding components to the same input
signal. For example, an single ink input signal might be processed
by both handwriting recognition and gesture recognition. Another is
the use of more than one recording device for the same input
(multiple microphones).</p>
<p>In order to make explicit these different kinds of multiple
interpretations and allow for concise statement of the annotations
associated with each, the <code>emma:one-of</code> element MAY
appear within another <code>emma:one-of</code> element. If
<code>emma:one-of</code> elements are nested then they MUST
indicate the kind of disjunction using the attribute
<code>disjunction-type</code>. The values of
<code>disjunction-type</code> are <code>{recognition,
understanding, multi-device, and multi-process}</code>. For the
most common use case, where there are multiple recognition results
and some of them have multiple interpretations, the top-level
<code>emma:one-of</code> is
<code>disjunction-type="recognition"</code> and the embedded
<code>emma:one-of</code> has the attribute
<code>disjunction-type="understanding"</code>.</p>
<p>As an example, in an interactive flight reservation application,
recognition yielded 'Boston' or 'Austin' and each had a semantic
interpretation as either the assertion of city name or the
specification of a flight query with the city as the destination,
this would be represented as follows in EMMA:</p>
<pre class="example">
<span>
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:one-of disjunction-type="recognition"
      start="12457990" end="12457995"
      <span>emma:medium="acoustic" emma:mode="voice"</span>&gt;
     &lt;emma:one-of disjunction-type="understanding"
         emma:tokens="boston"&gt;
       &lt;emma:interpretation&gt;
          &lt;assert&gt;&lt;city&gt;boston&lt;/city&gt;&lt;/assert&gt;
       &lt;/emma:interpretation&gt;
       &lt;emma:interpretation&gt;
          &lt;flight&gt;&lt;dest&gt;&lt;city&gt;boston&lt;/city&gt;&lt;/dest&gt;&lt;/flight&gt;
       &lt;/emma:interpretation&gt;
     &lt;/emma:one-of&gt;
     &lt;emma:one-of disjunction-type="understanding"
         emma:tokens="austin"&gt;
       &lt;emma:interpretation&gt;
          &lt;assert&gt;&lt;city&gt;austin&lt;/city&gt;&lt;/assert&gt;
       &lt;/emma:interpretation&gt;
       &lt;emma:interpretation&gt;
          &lt;flight&gt;&lt;dest&gt;&lt;city&gt;austin&lt;/city&gt;&lt;/dest&gt;&lt;/flight&gt;
       &lt;/emma:interpretation&gt;
     &lt;/emma:one-of&gt;
  &lt;/emma:one-of&gt;
&lt;/emma:emma&gt;
</span>
</pre>
<p>EMMA MAY explicitly represent ambiguity resulting from different
processes, devices, or sources using embedded
<code>emma:one-of</code> and the <code>disjunction-type</code>
attribute. Multiple different interpretations resulting from
different factors MAY also be listed within a single unstructured
<code>emma:one-of</code> though in this case it is more complex or
impossible to uncover the sources of the ambiguity if required by
later stages of processing. If there is no embedding in
<code>emma:one-of</code>, then the <code>disjunction-type</code>
attribute is not required. If the <code>disjunction-type</code>
attribute is missing then by default the source of disjunction is
unspecified.</p>
<p>The example case above could also be represented as:</p>
<pre class="example">
<span>
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:one-of  start="12457990" end="12457995"
<span>         emma:medium="acoustic" emma:mode="voice"</span>&gt;
     &lt;emma:interpretation emma:tokens="boston"&gt;
        &lt;assert&gt;&lt;city&gt;boston&lt;/city&gt;&lt;/assert&gt;
     &lt;/emma:interpretation&gt;
     &lt;emma:interpretation &gt;
        &lt;flight&gt;&lt;dest&gt;&lt;city&gt;boston&lt;/city&gt;&lt;/dest&gt;&lt;/flight&gt;
     &lt;/emma:interpretation&gt;
     &lt;emma:interpretation emma:tokens="austin"&gt;
        &lt;assert&gt;&lt;city&gt;austin&lt;/city&gt;&lt;/assert&gt;
     &lt;/emma:interpretation&gt;
     &lt;emma:interpretation emma:tokens="austin"&gt;
        &lt;flight&gt;&lt;dest&gt;&lt;city&gt;austin&lt;/city&gt;&lt;/dest&gt;&lt;/flight&gt;
     &lt;/emma:interpretation&gt;
  &lt;/emma:one-of&gt;
&lt;/emma:emma&gt;
</span>
</pre>
<p>But in this case information about which interpretations
resulted from speech recognition and which resulted from language
understanding is lost.</p>
<p>A list of <code>emma:interpretation</code> elements within an
<code>emma:one-of</code> MUST be sorted best-first by some measure
of quality. The quality measure is <code>emma:confidence</code> if
present, otherwise, the quality metric is platform-specific.</p>
<p>With embedded <code>emma:one-of</code> structures there is no
requirement for the confidence scores within different
<code>emma:one-of</code> to be on the same scale. For example, the
scores assigned by handwriting recognition might not be comparable
to those assigned by gesture recognition. Similarly, if multiple
recognizers are used there is no guarantee that their confidence
scores will be comparable. For this reason the ordering requirement
on <code>emma:interpretation</code> within <code>emma:one-of</code>
only applies locally to sister <code>emma:interpretation</code>
elements within each <code>emma:one-of</code>. There is no
requirement on the ordering of embedded <code>emma:one-of</code>
elements within a higher <code>emma:one-of</code> element.</p>
<p>While <code>emma:medium</code> and <code>emma:mode</code> are
optional on <code>emma:one-of</code>, note that all EMMA
interpretations must be annotated for <code>emma:medium</code> and
<code>emma:mode</code>, so either these annotations must appear
directly on all of the contained <code>emma:interpretation</code>
elements within the <code>emma:one-of</code>, or they must appear
on the <code>emma:one-of</code> element itself, or they must appear
on an ancestor <code>emma:one-of</code> element, or they must
appear on an earlier stage of the derivation listed in
<code>emma:derivation</code>.</p>
<h3 id="s3.3.2">3.3.2 <code>emma:group</code> element</h3>
<table class="defn" summary="property definition" width="98%"
cellpadding="5" cellspacing="0">
<tbody>
<tr>
<th>Annotation</th>
<th>emma:group</th>
</tr>
<tr>
<th>Definition</th>
<td>A container element indicating that a number of interpretations
of distinct user inputs are grouped according to some
criteria.</td>
</tr>
<tr>
<th>Children</th>
<td>The <code>emma:group</code> element MUST immediately contain a
collection of one or more <code>emma:interpretation</code> elements
or container elements: <code>emma:one-of</code>,
<code>emma:group</code>, <code>emma:sequence</code> . It MAY also
contain an <span>optional single</span>
<code>emma:group-info</code> element. It MAY also contain
<span>multiple optional</span> <code>emma:derived-from</code>
element<span>s</span> and <span>an optional single</span>
<code>emma:info</code> <span>element</span>.</td>
</tr>
<tr>
<th>Attributes</th>
<td>
<ul>
<li><b>Required</b>: Attribute <code>id</code> of type
<code>xsd:ID</code></li>
<li><b>Optional</b>: The annotation attributes:
<code>emma:tokens</code>, <code>emma:process</code>,
<code>emma:lang</code>, <code>emma:signal</code>,
<code><span>emma:signal-size</span></code>,
<code>emma:media-type</code>, <code>emma:confidence</code>,
<code>emma:source</code>, <code>emma:start</code>,
<code>emma:end</code>, <code>emma:time-ref-uri</code>,
<code>emma:time-ref-anchor-point</code>,
<code>emma:offset-to-start</code>, <code>emma:duration</code>,
<code>emma:medium</code>, <code>emma:mode</code>,
<code>emma:function</code>, <code>emma:verbal</code>,
<code>emma:cost</code>, <code>emma:grammar-ref</code>,
<code>emma:endpoint-info-ref</code>, <code>emma:model-ref</code>,
<code>emma:dialog-turn</code>.</li>
</ul>
</td>
</tr>
<tr>
<th>Applies to</th>
<td>The <code>emma:group</code> element is legal only as a child of
<code>emma:emma</code>, <code>emma:one-of</code>,
<code>emma:group</code>, <code>emma:sequence</code>, or
<code>emma:derivation</code>.</td>
</tr>
</tbody>
</table>
<p>The <code>emma:group</code> element is used to indicate that the
contained interpretations are from distinct user inputs that are
related in some manner. <code>emma:group</code> MUST NOT be used
for containing the multiple stages of processing of a single user
input. Those MUST be contained in the <code>emma:derivation</code>
element instead <span>(<a href="#s4.1.2">Section 4.1.2</a>)</span>.
For groups of inputs in temporal order the more specialized
container <code>emma:sequence</code> MUST be used <span>(<a href=
"#s3.3.3">Section 3.3.3</a>)</span>. The following example shows
three interpretations derived from the speech input "Move this
ambulance here" and the tactile input related to two consecutive
points on a map.</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:group id="grp"
      emma:start="1087995961542"
      emma:end="1087995964542"&gt;
    &lt;emma:interpretation id="int1"
      <span>emma:medium="acoustic" emma:mode="voice"</span>&gt;
      &lt;action&gt;move&lt;/action&gt;
      &lt;object&gt;ambulance&lt;/object&gt;
      &lt;destination&gt;here&lt;/destination&gt;
    &lt;/emma:interpretation&gt;

    &lt;emma:interpretation id="int2"
      <span>emma:medium="tactile" emma:mode="ink"</span>&gt;
      &lt;x&gt;0.253&lt;/x&gt;
      &lt;y&gt;0.124&lt;/y&gt;
    &lt;/emma:interpretation&gt;

    &lt;emma:interpretation id="int3"
      <span>emma:medium="tactile" emma:mode="ink"</span>&gt;
      &lt;x&gt;0.866&lt;/x&gt;
      &lt;y&gt;0.724&lt;/y&gt;
    &lt;/emma:interpretation&gt;
  &lt;/emma:group&gt;
&lt;/emma:emma&gt;

</pre>
<p>The <code>emma:one-of</code> and <code>emma:group</code>
containers MAY be nested arbitrarily.</p>
<h4 id="s3.3.2.1">3.3.2.1 Indirect grouping criteria:
<code>emma:group-info</code> element</h4>
<table class="defn" summary="property definition" width="98%"
cellpadding="5" cellspacing="0">
<tbody>
<tr>
<th>Annotation</th>
<th>emma:group-info</th>
</tr>
<tr>
<th>Definition</th>
<td>The <code>emma:group-info</code> element contains or references
criteria used in establishing the grouping of interpretations in an
<code>emma:group</code> element.</td>
</tr>
<tr>
<th>Children</th>
<td>The <code>emma:group-info</code> element MUST either
immediately contain inline instance data specifying grouping
criteria or have the attribute <code>ref</code> referencing the
criteria.</td>
</tr>
<tr>
<th>Attributes</th>
<td>
<ul>
<li><b>Optional</b>: <code>ref</code> of type
<code>xsd:anyURI</code> referencing the grouping criteria;
alternatively the criteria MAY be provided inline as the content of
the <code>emma:group-info</code> element.</li>
</ul>
</td>
</tr>
<tr>
<th>Applies to</th>
<td>The <code>emma:group-info</code> element is legal only as a
child of <code>emma:group</code>.</td>
</tr>
</tbody>
</table>
<p>Sometimes it may be convenient to indirectly associate a given
group with information, such as grouping criteria. The
<code>emma:group-info</code> element might be used to make explicit
the criteria by which members of a group are associated. In the
following example, a group of two points is associated with a
description of grouping criteria based upon a sliding temporal
window of two seconds duration.</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"
    xmlns:ex="http://www.example.com/ns/group"&gt;
  &lt;emma:group id="grp"&gt;
    &lt;emma:group-info&gt;
      &lt;ex:mode&gt;temporal&lt;/ex:mode&gt;
      &lt;ex:duration&gt;2s&lt;/ex:duration&gt;
    &lt;/emma:group-info&gt;

    &lt;emma:interpretation id="int1"
<span>      emma:medium="tactile" emma:mode="ink"</span>&gt;
      &lt;x&gt;0.253&lt;/x&gt;
      &lt;y&gt;0.124&lt;/y&gt;
    &lt;/emma:interpretation&gt;

    &lt;emma:interpretation id="int2"
      <span>emma:medium="tactile" emma:mode="ink"</span>&gt;
      &lt;x&gt;0.866&lt;/x&gt;
      &lt;y&gt;0.724&lt;/y&gt;
    &lt;/emma:interpretation&gt;
  &lt;/emma:group&gt;
&lt;/emma:emma&gt;
</pre>
<p>You might also use <code>emma:group-info</code> to refer to a
named grouping criterion using external reference, for
instance:</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"
    xmlns:ex="http://www.example.com/ns/group"&gt;
  &lt;emma:group id="grp"&gt;
    &lt;emma:group-info ref="http://www.example.com/criterion42"/&gt;
    &lt;emma:interpretation id="int1"
      <span>emma:medium="tactile" emma:mode="ink"</span>&gt;
      &lt;x&gt;0.253&lt;/x&gt;
      &lt;y&gt;0.124&lt;/y&gt;
    &lt;/emma:interpretation&gt;

    &lt;emma:interpretation id="int2"
      <span>emma:medium="tactile" emma:mode="ink"</span>&gt;
      &lt;x&gt;0.866&lt;/x&gt;
      &lt;y&gt;0.724&lt;/y&gt;
    &lt;/emma:interpretation&gt;
  &lt;/emma:group&gt;
&lt;/emma:emma&gt;
</pre>
<h3 id="s3.3.3">3.3.3 <code>emma:sequence</code> element</h3>
<table class="defn" summary="property definition" width="98%"
cellpadding="5" cellspacing="0">
<tbody>
<tr>
<th>Annotation</th>
<th>emma:sequence</th>
</tr>
<tr>
<th>Definition</th>
<td>A container element indicating that a number of interpretations
of distinct user inputs are in temporal sequence.</td>
</tr>
<tr>
<th>Children</th>
<td>The <code>emma:sequence</code> element MUST immediately contain
a collection of one or more <code>emma:interpretation</code>
elements or container elements: <code>emma:one-of</code>,
<code>emma:group</code>, <code>emma:sequence</code> . It MAY also
contain <span>multiple optional</span>
<code>emma:derived-from</code> element<span>s</span> and <span>an
optional single</span> <code>emma:info</code>
<span>element</span>.</td>
</tr>
<tr>
<th>Attributes</th>
<td>
<ul>
<li><b>Required</b>: Attribute <code>id</code> of type
<code>xsd:ID</code></li>
<li><b>Optional</b>: The annotation attributes:
<code>emma:tokens</code>, <code>emma:process</code>,
<code>emma:lang</code>, <code>emma:signal</code>,
<code><span>emma:signal-size</span></code>,
<code>emma:media-type</code>, <code>emma:confidence</code>,
<code>emma:source</code>, <code>emma:start</code>,
<code>emma:end</code>, <code>emma:time-ref-uri</code>,
<code>emma:time-ref-anchor-point</code>,
<code>emma:offset-to-start</code>, <code>emma:duration</code>,
<code>emma:medium</code>, <code>emma:mode</code>,
<code>emma:function</code>, <code>emma:verbal</code>,
<code>emma:cost</code>, <code>emma:grammar-ref</code>,
<code>emma:endpoint-info-ref</code>, <code>emma:model-ref</code>,
<code>emma:dialog-turn</code>.</li>
</ul>
</td>
</tr>
<tr>
<th>Applies to</th>
<td>The <code>emma:sequence</code> element is legal only as a child
of <code>emma:emma</code>, <code>emma:one-of</code>,
<code>emma:group</code>, <code>emma:sequence</code>, or
<code>emma:derivation</code>.</td>
</tr>
</tbody>
</table>
<p>The <code>emma:sequence</code> element is used to indicate that
the contained interpretations are sequential in time, as in the
following example, which indicates that two points made with a pen
are in temporal order.</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:sequence id="seq1"&gt;
    &lt;emma:interpretation id="int1"
        <span>emma:medium="tactile"</span> emma:mode="ink"&gt;
      &lt;x&gt;0.253&lt;/x&gt;
      &lt;y&gt;0.124&lt;/y&gt;
    &lt;/emma:interpretation&gt;

    &lt;emma:interpretation id="int2"
        <span>emma:medium="tactile"</span> emma:mode="ink"&gt;
      &lt;x&gt;0.866&lt;/x&gt;
      &lt;y&gt;0.724&lt;/y&gt;
    &lt;/emma:interpretation&gt;
  &lt;/emma:sequence&gt;
&lt;/emma:emma&gt;
</pre>
<p>The <code>emma:sequence</code> container MAY be combined with
<code>emma:one-of</code> and <code>emma:group</code> in arbitrary
nesting structures. The order of children in the content of the
<code>emma:sequence</code> element corresponds to a sequence of
interpretations. This ordering does not imply any particular
definition of sequentiality. EMMA processors are expected therefore
to use the <code>emma:sequence</code> element to hold
interpretations which are either strictly sequential in nature
(e.g. the end-time of an interpretation precedes the start-time of
its follower), or which overlap in some manner (e.g. the start-time
of a follower interpretation precedes the end-time of its
precedent). It is possible to use timestamps to provide fine
grained annotation for the sequence of interpretations that are
sequential in time <span>(see <a href="#s4.2.10">Section
4.2.10)</a></span>.</p>
<p>In the following more complex example, a sequence of two pen
gestures in <code>emma:sequence</code> and a speech input in
<code>emma:interpretation</code> <span>is</span> contained in an
<code>emma:group</code>.</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:group id="grp"&gt;
     &lt;emma:interpretation id="int1" emma:medium="acoustic"
         emma:mode="voice"&gt;
       &lt;action&gt;move&lt;/action&gt;
       &lt;object&gt;this-battleship&lt;/object&gt;
       &lt;destination&gt;here&lt;/destination&gt;
     &lt;/emma:interpretation&gt;

     &lt;emma:sequence id="seq1"&gt;
       &lt;emma:interpretation id="int2" emma:medium="tactile"
           emma:mode="ink"&gt;
         &lt;x&gt;0.253&lt;/x&gt;
         &lt;y&gt;0.124&lt;/y&gt;
       &lt;/emma:interpretation&gt;

     &lt;emma:interpretation id="int3" emma:medium="tactile"
         emma:mode="ink"&gt;
       &lt;x&gt;0.866&lt;/x&gt;
       &lt;y&gt;0.724&lt;/y&gt;
     &lt;/emma:interpretation&gt;
   &lt;/emma:sequence&gt;
 &lt;/emma:group&gt;
&lt;/emma:emma&gt;
</pre>
<h3 id="s3.4">3.4 Lattice element</h3>
<p>In addition to providing the ability to represent N-best lists
of interpretations using <code>emma:one-of</code>, EMMA also
provides the capability to represent lattices of words or other
symbols using the <code>emma:lattice</code> element. Lattices
provide a compact representation of large lists of possible
recognition results or interpretations for speech, pen, or
multimodal inputs.</p>
<p>In addition to providing a representation for lattice output
from speech recognition, another important use case for lattices is
for representation of the results of gesture and handwriting
recognition from a pen modality component. Lattices can also be
used to compactly represent multiple possible meaning
representations. Another use case for the lattice representation is
for associating confidence scores and other annotations with
individual words within a speech recognition result string.</p>
<p>Lattices are compactly described by a list of transitions
between nodes. For each transition the start and end nodes MUST be
defined, along with the label for the transition. Initial and final
nodes MUST also be indicated. The following figure provides a
graphical representation of a speech recognition lattice which
compactly represents eight different sequences of words.</p>
<p><img alt="speech lattice" src="lattice.png" /></p>
<p>which expands to:</p>
<pre>
a. flights to boston from portland today please
b. flights to austin from portland today please
c. flights to boston from oakland today please
d. flights to austin from oakland today please
e. flights to boston from portland tomorrow
f. flights to austin from portland tomorrow
g. flights to boston from oakland tomorrow
h. flights to austin from oakland tomorrow
</pre>
<h4 id="s3.4.1">3.4.1 Lattice markup: <code>emma:lattice</code>,
<code>emma:arc</code>, <code>emma:node</code> elements</h4>
<table class="defn" summary="property definition" width="98%"
cellpadding="5" cellspacing="0">
<tbody>
<tr>
<th>Annotation</th>
<th>emma:lattice</th>
</tr>
<tr>
<th>Definition</th>
<td>An element which encodes a lattice representation of user
input.</td>
</tr>
<tr>
<th>Children</th>
<td>The <code>emma:lattice</code> element MUST immediately contain
one or more <code>emma:arc</code> elements and zero or more
<code>emma:node</code> elements.</td>
</tr>
<tr>
<th>Attributes</th>
<td>
<ul>
<li><b>Required</b>:
<ul>
<li><code>initial</code> <span>of type
<code>xsd:nonNegativeInteger</code></span> indicating the number of
the initial node of the lattice.</li>
<li><code>final</code> contains a space-separated list of
<code>xsd:nonNegativeInteger</code> indicating the numbers of the
final nodes in the lattice.</li>
</ul>
</li>
<li><b>Optional</b>: <code>emma:time-ref-uri</code>,
<code>emma:time-ref-anchor-point</code>.</li>
</ul>
</td>
</tr>
<tr>
<th>Applies to</th>
<td>The <code>emma:lattice</code> element is legal only as a child
of the <code>emma:interpretation</code> element.</td>
</tr>
<tr>
<th>Annotation</th>
<th>emma:arc</th>
</tr>
<tr>
<th>Definition</th>
<td>An element which encodes a transition between two nodes in a
lattice. The label associated with the arc in the lattice is
represented in the content of <code>emma:arc</code>.</td>
</tr>
<tr>
<th>Children</th>
<td>The <code>emma:arc</code> element MUST immediately contain
either character data or a single application namespace element or
be empty, in the case of epsilon transitions. It MAY contain an
<code>emma:info</code> element containing application or vendor
specific annotations.</td>
</tr>
<tr>
<th>Attributes</th>
<td>
<ul>
<li><b>Required</b>:
<ul>
<li><code>from</code> <span>of type
<code>xsd:nonNegativeInteger</code></span> indicating the number of
the starting node for the arc.</li>
<li><code>to</code> <span>of type
<code>xsd:nonNegativeInteger</code></span> indicating the number of
the ending node for the arc.</li>
</ul>
</li>
<li><b>Optional</b>: <code>emma:start</code>,
<code>emma:end</code>, <code>emma:offset-to-start</code>,
<code>emma:duration</code>, <code>emma:confidence</code>,
<code>emma:cost</code>, <code>emma:lang</code>,
<code>emma:medium</code>, <code>emma:mode</code>,
<code>emma:source</code>.</li>
</ul>
</td>
</tr>
<tr>
<th>Applies to</th>
<td>The <code>emma:arc</code> element is legal only as a child of
the <code>emma:lattice</code> element.</td>
</tr>
<tr>
<th>Annotation</th>
<th>emma:node</th>
</tr>
<tr>
<th>Definition</th>
<td>An element which represents a node in the lattice. The
<code>emma:node</code> elements are not required to describe a
lattice but might be added to provide a location for annotations on
nodes in a lattice. There MUST be at most one
<code>emma:node</code> specification for each numbered node in the
lattice.</td>
</tr>
<tr>
<th>Children</th>
<td>An OPTIONAL <code>emma:info</code> element for application or
vendor specific annotations on the node.</td>
</tr>
<tr>
<th>Attributes</th>
<td>
<ul>
<li><b>Required</b>:
<ul>
<li><code>node-number</code> <span>of type
<code>xsd:nonNegativeInteger</code></span> indicating the
<span>node number</span> in the lattice.</li>
</ul>
</li>
<li><b>Optional</b>: <code>emma:confidence</code>,
<code>emma:cost</code>.</li>
</ul>
</td>
</tr>
<tr>
<th>Applies to</th>
<td>The <code>emma:node</code> element is legal only as a child of
the <code>emma:lattice</code> element.</td>
</tr>
</tbody>
</table>
<p>In EMMA, a lattice is represented using an element
<code>emma:lattice</code>, which has attributes
<code>initial</code> and <code>final</code> for indicating the
initial and final nodes of the lattice. For the lattice
<span>below</span>, this will be: <code>&lt;emma:lattice
initial="1" final="8"/&gt;</code>. The nodes are numbered with
integers. If there is more than one distinct final node in the
lattice the nodes MUST be represented as a space separated list in
the value of the <code>final</code> attribute e.g.
<code>&lt;emma:lattice initial="1" final="9 10 23"/&gt;</code>.
There MUST only be one initial node in an EMMA lattice. Each
transition in the lattice is represented as an element
<code>emma:arc</code> with attributes <code>from</code> and
<code>to</code> which indicate the nodes where the transition
starts and ends. The arc's label is represented as the content of
the <code>emma:arc</code> element and MUST be any well-formed
character or XML content. In the example here the contents are
words. Empty (epsilon) transitions in a lattice MUST be represented
in the <code>emma:lattice</code> representation as
<code>emma:arc</code> <span>empty</span> elements, e.g.
<code>&lt;emma:arc from="1" to="8"/&gt;</code>.</p>
<p>The example speech lattice above would be represented in EMMA
markup as follows:</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:interpretation id="interp1"
    <span>emma:medium="acoustic" emma:mode="voice"</span>&gt;
    &lt;emma:lattice initial="1" final="8"&gt;
      &lt;emma:arc from="1" to="2"&gt;flights&lt;/emma:arc&gt;

      &lt;emma:arc from="2" to="3"&gt;to&lt;/emma:arc&gt;
      &lt;emma:arc from="3" to="4"&gt;boston&lt;/emma:arc&gt;
      &lt;emma:arc from="3" to="4"&gt;austin&lt;/emma:arc&gt;
      &lt;emma:arc from="4" to="5"&gt;from&lt;/emma:arc&gt;

      &lt;emma:arc from="5" to="6"&gt;portland&lt;/emma:arc&gt;
      &lt;emma:arc from="5" to="6"&gt;oakland&lt;/emma:arc&gt;
      &lt;emma:arc from="6" to="7"&gt;today&lt;/emma:arc&gt;
      &lt;emma:arc from="7" to="8"&gt;please&lt;/emma:arc&gt;

      &lt;emma:arc from="6" to="8"&gt;tomorrow&lt;/emma:arc&gt;
    &lt;/emma:lattice&gt;
  &lt;/emma:interpretation&gt;
&lt;/emma:emma&gt;
</pre>
<p>Alternatively, if we wish to represent the same information as
an N-best list using <code>emma:one-of,</code> we would have the
more verbose representation:</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:one-of id="nbest1" <span>emma:medium="acoustic" emma:mode="voice"</span>&gt;
    &lt;emma:interpretation id="interp1"&gt;
      &lt;text&gt;flights to boston from portland today please&lt;/text&gt;
    &lt;/emma:interpretation&gt;

    &lt;emma:interpretationid="interp2"&gt;
      &lt;text&gt;flights to boston from portland tomorrow&lt;/text&gt;
    &lt;/emma:interpretation&gt;

    &lt;emma:interpretation id="interp3"&gt;
      &lt;text&gt;flights to austin from portland today please&lt;/text&gt;
    &lt;/emma:interpretation&gt;

    &lt;emma:interpretation id="interp4"&gt;
      &lt;text&gt;flights to austin from portland tomorrow&lt;/text&gt;
    &lt;/emma:interpretation&gt;

    &lt;emma:interpretation id="interp5"&gt;
      &lt;text&gt;flights to boston from oakland today please&lt;/text&gt;
    &lt;/emma:interpretation&gt;

    &lt;emma:interpretation id="interp6"&gt;
      &lt;text&gt;flights to boston from oakland tomorrow&lt;/text&gt;
    &lt;/emma:interpretation&gt;

    &lt;emma:interpretation id="interp7"&gt;
      &lt;text&gt;flights to austin from oakland today please&lt;/text&gt;
    &lt;/emma:interpretation&gt;

    &lt;emma:interpretation id="interp8"&gt;
      &lt;text&gt;flights to austin from oakland tomorrow&lt;/text&gt;
    &lt;/emma:interpretation&gt;
  &lt;/emma:one-of&gt;
&lt;/emma:emma&gt;
</pre>
<p>The lattice representation avoids the need to enumerate all of
the possible word sequences. Also, as detailed below, the
<code>emma:lattice</code> representation enables placement of
annotations on individual words in the input.</p>
<p>For use cases involving the representation of gesture/ink
lattices and use cases involving lattices of semantic
interpretations, EMMA allows for application namespace elements to
appear within <code>emma:arc</code>.</p>
<p>For example a sequence of two gestures, each of which is
recognized as either a line or a circle<span>,</span> might be
represented as follows:</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:interpretation id="interp1"
    <span>emma:medium="acoustic" emma:mode="voice"</span>&gt;
    &lt;emma:lattice initial="1" final="3"&gt;
      &lt;emma:arc from="1" to="2"&gt;
        &lt;circle radius="100"/&gt;
      &lt;/emma:arc&gt;
      &lt;emma:arc from="2" to="3"&gt;
        &lt;line length="628"/&gt;
      &lt;/emma:arc&gt;
      &lt;emma:arc from="1" to="2"&gt;
        &lt;circle radius="200"/&gt;
      &lt;/emma:arc&gt;
      &lt;emma:arc from="2" to="3"&gt;
        &lt;line length="1256"/&gt;
      &lt;/emma:arc&gt;
    &lt;/emma:lattice&gt;
  &lt;/emma:interpretation&gt;
&lt;/emma:emma&gt;
</pre>
<p>As an example of a lattice of semantic interpretations, in a
travel application where the source is either "Boston" or
"Austin"and the destination is either "Newark" or "New York", the
possibilities might be represented in a lattice as follows:</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:interpretation id="interp1"
    <span>emma:medium="acoustic" emma:mode="voice"</span>&gt;
    &lt;emma:lattice initial="1" final="3"&gt;
      &lt;emma:arc from="1" to="2"&gt;
        &lt;source city="boston"/&gt;
      &lt;/emma:arc&gt;
      &lt;emma:arc from="2" to="3"&gt;
        &lt;destination city="newark"/&gt;
      &lt;/emma:arc&gt;
      &lt;emma:arc from="1" to="2"&gt;
        &lt;source city="austin"/&gt;
      &lt;/emma:arc&gt;
      &lt;emma:arc from="2" to="3"&gt;
        &lt;destination city="new york"/&gt;
      &lt;/emma:arc&gt;
    &lt;/emma:lattice&gt;
  &lt;/emma:interpretation&gt;
&lt;/emma:emma&gt;
</pre>
<p>The <code>emma:arc</code> element MAY contain either an
application namespace element or character data. It MUST NOT
contain combinations of application namespace elements and
character data. However, an <code>emma:info</code> element MAY
appear within an <code>emma:arc</code> element alongside character
data, in order to allow for the association of vendor or
application specific annotations on a single word or symbol in a
lattice.</p>
<p>So, in summary, there are four groupings of content that can
appear within <code>emma:arc</code>:</p>
<ul>
<li>Character Data e.g. a recognized word in a speech lattice.</li>
<li>Character Data and a single <code>emma:info</code> element
providing vendor or application specific annotations that apply to
the character data.</li>
<li>An application namespace element e.g. the gesture and
<span>semantic interpretation</span> lattice examples above.</li>
<li>An application namespace element and a single
<code>emma:info</code> element providing vendor or application
specific annotations that apply to the character data.</li>
</ul>
<h4 id="s3.4.2">3.4.2 Annotations on lattices</h4>
<p>The encoding of lattice arcs as XML elements
(<code>emma:arc</code>) enables arcs to be annotated with metadata
such as timestamps, costs, or confidence scores:</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:interpretation id="interp1"
    <span>emma:medium="acoustic" emma:mode="voice"</span>&gt;
    &lt;emma:lattice initial="1" final="8"&gt;
      &lt;emma:arc
       from="1"
       to="2"
       emma:start="1087995961542"
       emma:end="1087995962042"
       emma:cost="30"&gt;
         flights
      &lt;/emma:arc&gt;

      &lt;emma:arc
       from="2"
       to="3"
       emma:start="1087995962042"
       emma:end="1087995962542"
       emma:cost="20"&gt;
         to
      &lt;/emma:arc&gt;

      &lt;emma:arc
       from="3"
       to="4"
       emma:start="1087995962542"
       emma:end="1087995963042"
       emma:cost="50"&gt;
         boston
      &lt;/emma:arc&gt;

      &lt;emma:arc
       from="3"
       to="4"
       emma:start="1087995963042"
       emma:end="1087995963742"
       emma:cost="60"&gt;
         austin
      &lt;/emma:arc&gt;
      ...
    &lt;/emma:lattice&gt;
  &lt;/emma:interpretation&gt;
&lt;/emma:emma&gt;
</pre>
<p>The following EMMA attributes MAY be placed on
<code>emma:arc</code> elements: absolute timestamps
(<code>emma:start</code>, <code>emma:end</code>), relative
timestamps ( <code>emma:offset-to-start</code>,
<code>emma:duration</code>), <code>emma:confidence</code>,
<code>emma:cost</code>, the human language of the input
(<code>emma:lang</code>), <code>emma:medium</code>,
<code>emma:mode</code>, and <code>emma:source</code>. The use case
for <code>emma:medium</code>, <code>emma:mode</code>, and
<code>emma:source</code> is for lattices which contains content
from different input modes. The <code>emma:arc</code> element MAY
also contain an <code>emma:info</code> element for specification of
vendor and application specific annotations on the arc.</p>
<p>The timestamps that appear on <code>emma:arc</code> elements do
not necessarily indicate the start and end of the arc itself. They
MAY indicate the start and end of the signal corresponding to the
label on the arc. As a result there is no requirement that the
<code>emma:end</code> timestamp on an arc going into a node should
be equivalent to the <code>emma:start</code> of all arcs going out
of that node. Furthermore there is no guarantee that the left to
right order of arcs in a lattice will correspond to the temporal
order of the input signal. The lattice representation is an
abstraction that represents a range of possible interpretations of
a user's input and is not intended to necessarily be a
representation of temporal order.</p>
<p>Costs are typically application and device dependent. There are
a variety of ways that individual arc costs might be combined to
produce costs for specific paths through the lattice. This
specification does not standardize the way for these costs to be
combined; it is up to the applications and devices to determine how
such derived costs would be computed and used.</p>
<p>For some lattice formats, it is also desirable to annotate the
nodes in the lattice themselves with information such as costs. For
example in speech recognition, costs might be placed on nodes as a
result of word penalties or redistribution of costs. For this
purpose EMMA also provides an <code>emma:node</code> element which
can host annotations such as <code>emma:cost</code>. The
<code>emma:node</code> element MUST have an attribute
<code>node-number</code> which indicates the number of the node.
There MUST be at most one <code>emma:node</code> specification for
a given numbered node in the lattice. In our example, if there was
a cost of <b>100</b> on the final state this could be represented
as follows:</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:interpretation id="interp1" 
    <span>emma:medium="acoustic" emma:mode="voice"</span>&gt;
    &lt;emma:lattice initial="1" final="8"&gt;
      &lt;emma:arc
       from="1"
       to="2"
       emma:start="1087995961542"
       emma:end="1087995962042"
       emma:cost="30"&gt;
         flights
      &lt;/emma:arc&gt;
      &lt;emma:arc
       from="2"
       to="3"
       emma:start="1087995962042"
       emma:end="1087995962542"
       emma:cost="20"&gt;
         to
      &lt;/emma:arc&gt;

      &lt;emma:arc
       from="3"
       to="4"
       emma:start="1087995962542"
       emma:end="1087995963042"
       emma:cost="50"&gt;
         boston
      &lt;/emma:arc&gt;
      &lt;emma:arc
       from="3"
       to="4"
       emma:start="1087995963042"
       emma:end="1087995963742"
       emma:cost="60"&gt;
         austin
      &lt;/emma:arc&gt;
        ...
      &lt;emma:node node-number="8" emma:cost="100"/&gt;
    &lt;/emma:lattice&gt;
  &lt;/emma:interpretation&gt;
&lt;/emma:emma&gt;
</pre>
<h4 id="s3.4.3">3.4.3 Relative timestamps on lattices</h4>
<p>The relative timestamp mechanism in EMMA is intended to provide
temporal information about arcs in a lattice in relative terms
using offsets in milliseconds. In order to do this the absolute
time MAY be specified on <code>emma:interpretation</code>; both
<code>emma:time-ref-uri</code> and
<code>emma:time-ref-anchor-point</code> apply to
<code>emma:lattice</code> and MAY be used there to set the anchor
point for offsets to the start of the absolute time specified on
<code>emma:interpretation</code>. The offset in milliseconds to the
beginning of each arc MAY then be indicated on each
<code>emma:arc</code> in the <code>emma:offset-to-start</code>
attribute.</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;

  &lt;emma:interpretation id="interp1"
          emma:start="1087995961542" emma:end="1087995963042"
          <span>emma:medium="acoustic" emma:mode="voice"</span>&gt;
    &lt;emma:lattice emma:time-ref-uri="#interp1"
        emma:time-ref-anchor-point="start"
        initial="1" final="4"&gt;
      &lt;emma:arc
       from="1"
       to="2"
       emma:offset-to-start="0"&gt;
         flights
      &lt;/emma:arc&gt;
      &lt;emma:arc
       from="2"
       to="3"
       emma:offset-to-start="500"&gt;
         to
      &lt;/emma:arc&gt;

      &lt;emma:arc
       from="3"
       to="4"
       emma:offset-to-start="1000"&gt;
         boston
      &lt;/emma:arc&gt;
    &lt;/emma:lattice&gt;
  &lt;/emma:interpretation&gt;
&lt;/emma:emma&gt;
</pre>
<p>Note that the offset for the first <code>emma:arc</code> MUST
always be zero since the EMMA attribute
<code>emma:offset-to-start</code> indicates the number of
milliseconds from the anchor point to the <i>start</i> of the piece
of input associated with the <code>emma:arc</code>, in this case
the word "flights".</p>
<h3 id="s3.5">3.5 Literal semantics: <code>emma:literal</code>
element</h3>
<table class="defn" summary="property definition" width="98%"
cellpadding="5" cellspacing="0">
<tbody>
<tr>
<th>Annotation</th>
<th>emma:literal</th>
</tr>
<tr>
<th>Definition</th>
<td>An element that contains string literal output.</td>
</tr>
<tr>
<th>Children</th>
<td>String literal</td>
</tr>
<tr>
<th>Attributes</th>
<td>None.</td>
</tr>
<tr>
<th>Applies to</th>
<td>The <code>emma:literal</code> is a child of
<code>emma:interpretation</code>.</td>
</tr>
</tbody>
</table>
<p>Certain EMMA processing components produce semantic results in
the form of string literals without any surrounding application
namespace markup. These MUST be placed with the EMMA element
<code>emma:literal</code> within <code>emma:interpretation</code>.
For example, if a semantic interpreter simply returned "boston"
this could be represented in EMMA as:</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:interpretation <span>id="r1" <br />
     emma:medium="acoustic" emma:mode="voice"</span>&gt;
    &lt;emma:literal&gt;boston&lt;/emma:literal&gt;
  &lt;/emma:interpretation&gt;
&lt;/emma:emma&gt;
</pre>
<p>Note that a raw recognition result of a sequence of words from
speech recognition is also a kind of string literal and can be
contained within <code>emma:literal</code>. For example,
recognition of the string "flights to san francisco" can be
represented in EMMA as follows:</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:interpretation <span>id="r1" <br />
     emma:medium="acoustic" emma:mode="voice"</span>&gt;
    &lt;emma:literal&gt;flights to san francisco&lt;/emma:literal&gt;
  &lt;/emma:interpretation&gt;
&lt;/emma:emma&gt;
</pre>
<h2 id="s4">4. EMMA annotations</h2>
<p>This section defines annotations in the EMMA namespace including
both attributes and elements. The values are specified in terms of
the data types defined by XML Schema Part 2: Datatypes <span>Second
Edition</span> [<a href="#XSD2"><span>XML Schema
Datatypes</span></a>].</p>
<h3 id="s4.1">4.1 EMMA annotation elements</h3>
<h4 id="s4.1.1">4.1.1 Data model: <code>emma:model</code>
element</h4>
<table class="defn" summary="property definition" width="98%"
cellpadding="5" cellspacing="0">
<tbody>
<tr>
<th>Annotation</th>
<th>emma:model</th>
</tr>
<tr>
<th>Definition</th>
<td>The <code>emma:model</code> either references or provides
inline the data model for the instance data.</td>
</tr>
<tr>
<th>Children</th>
<td>If a <code>ref</code> attribute is not specified then this
element contains the data model inline.</td>
</tr>
<tr>
<th>Attributes</th>
<td>
<ul>
<li><b>Required</b>:
<ul>
<li><code>id</code> of type <code>xsd:ID</code>.</li>
</ul>
</li>
<li><b>Optional</b>:
<ul>
<li><code>ref</code> of type <code>xsd:anyURI</code> that
references the data model. Note that either an <code>ref</code>
attribute or in-line data model (but not both) MUST be
specified.</li>
</ul>
</li>
</ul>
</td>
</tr>
<tr>
<th>Applies to</th>
<td>The <code>emma:model</code> element MAY appear only as a child
of <code>emma:emma</code>.</td>
</tr>
</tbody>
</table>
<p>The data model that may be used to express constraints on the
structure and content of instance data is specified as one of the
annotations of the instance. Specifying the data model is OPTIONAL,
in which case the data model can be said to be implicit. Typically
the data model is pre-established by the application.</p>
<p>The data model is specified with the <code>emma:model</code>
annotation defined as an element in the EMMA namespace. If the data
model for the contents of a <code>emma:interpretation</code>,
container elements, or application namespace element is to be
specified in EMMA, the attribute <code>emma:model-ref</code> MUST
be specified on the <code>emma:interpretation</code>, container
element, or application namespace element. Note that since multiple
<code>emma:model</code> elements might be specified under the
<code>emma:emma</code> it is possible to refer to multiple data
models within a single EMMA document. For example, different
alternative interpretations under an <code>emma:one-of</code> might
have different data models. In this case, an
<code>emma:model-ref</code> attribute would appear on each
<code>emma:interpretation</code> element in the N-best list with
its value being the <code>id</code> of the <code>emma:model</code>
element for that particular interpretation.</p>
<p>The data model is closely related to the interpretation data,
and is typically specified as the annotation related to the
<code>emma:interpretation</code> or <code>emma:one-of</code>
elements.</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:model id="model1" ref="http://example.com/models/city.xml"/&gt;
  &lt;emma:interpretation id="int1" emma:model-ref="model1"
    <span>emma:medium="acoustic" emma:mode="voice"</span>&gt;
    &lt;city&gt; London &lt;/city&gt;
    &lt;country&gt; UK &lt;/country&gt;
  &lt;/emma:interpretation&gt;
&lt;/emma:emma&gt;
</pre>
<p>The <code>emma:model</code> annotation MAY reference any element
or attribute in the application instance data, as well as any EMMA
container element (<code>emma:one-of</code>,
<code>emma:group</code>, or <code>emma:sequence</code>).</p>
<p>The data model annotation MAY be used to either reference an
external data model with the <code>ref</code> attribute or provide
a data model as in-line content. Either a <code>ref</code>
attribute or in-line data model (but not both) MUST be
specified.</p>
<h4 id="s4.1.2">4.1.2 Interpretation derivation:
<code>emma:derived-from</code> element and
<code>emma:derivation</code> element</h4>
<table class="defn" summary="property definition" width="98%"
cellpadding="5" cellspacing="0">
<tbody>
<tr>
<th>Annotation</th>
<th>emma:derived-from</th>
</tr>
<tr>
<th>Definition</th>
<td>An empty element which provides a reference to the
interpretation which the element it appears on was derived
from.</td>
</tr>
<tr>
<th>Children</th>
<td>None</td>
</tr>
<tr>
<th>Attributes</th>
<td>
<ul>
<li><b>Required</b>:
<ul>
<li><code>resource</code> of type <code>xsd:anyURI</code> that
references the interpretation from which the current interpretation
is derived.</li>
</ul>
</li>
<li><b>Optional</b>:
<ul>
<li><code>composite</code> of type <code>xsd:boolean</code> that is
<code>"true"</code> if the derivation step combines multiple inputs
and <code>"false"</code> if not. If <code>composite</code> is not
specified the value is <code>"false"</code> by default.</li>
</ul>
</li>
</ul>
</td>
</tr>
<tr>
<th>Applies to</th>
<td>The <code>emma:derived-from</code> element is legal only as a
child of <code>emma:interpretation</code>,
<code>emma:one-of</code>, <code>emma:group</code>, or
<code>emma:sequence</code>.</td>
</tr>
<tr>
<th>Annotation</th>
<th>emma:derivation</th>
</tr>
<tr>
<th>Definition</th>
<td>An element which contains interpretation and container elements
representing earlier stages in the processing of the input.</td>
</tr>
<tr>
<th>Children</th>
<td>One or more <code>emma:interpretation</code>,
<code>emma:one-of</code>, <code>emma:sequence</code>, or
<code>emma:group</code> elements.</td>
</tr>
<tr>
<th>Attributes</th>
<td>None</td>
</tr>
<tr>
<th>Applies to</th>
<td>The <code>emma:derivation</code> MAY appear only as a child of
the <code>emma:emma</code> element.</td>
</tr>
</tbody>
</table>
<p>Instances of interpretations are in general derived from other
instances of interpretation in a process that goes from raw data to
increasingly refined representations of the input. The derivation
annotation is used to link any two interpretations that are related
by representing the source and the outcome of an interpretation
process. For instance, a speech recognition process can return the
following result in the form of raw text:</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:interpretation id="raw"<br />
    <span>emma:medium="acoustic" emma:mode="voice"</span>&gt;
    &lt;answer&gt;From Boston to Denver tomorrow&lt;/answer&gt;
  &lt;/emma:interpretation&gt;

&lt;/emma:emma&gt;
</pre>
<p>A first interpretation process will produce:</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:interpretation id="better"<br />
    <span>emma:medium="acoustic" emma:mode="voice"</span>&gt;
    &lt;origin&gt;Boston&lt;/origin&gt;
    &lt;destination&gt;Denver&lt;/destination&gt;
    &lt;date&gt;tomorrow&lt;/date&gt;
  &lt;/emma:interpretation&gt;
&lt;/emma:emma&gt;
</pre>
<p>A second interpretation process, aware of the current date, will
be able to produce a more refined instance, such as:</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:interpretation id="best"
    <span>emma:medium="acoustic" emma:mode="voice"</span>&gt;
    &lt;origin&gt;Boston&lt;/origin&gt;
    &lt;destination&gt;Denver&lt;/destination&gt;
    &lt;date&gt;20030315&lt;/date&gt;
  &lt;/emma:interpretation&gt;
&lt;/emma:emma&gt;
</pre>
<p>The interaction manager might need to have access to the three
levels of interpretation. The <code>emma:derived-from</code>
annotation element can be used to establish a chain of derivation
relationships as in the following example:</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:derivation&gt;
    &lt;emma:interpretation id="raw"<br />
<span>      emma:medium="acoustic" emma:mode="voice"</span>&gt;
      &lt;answer&gt;From Boston to Denver tomorrow&lt;/answer&gt;
    &lt;/emma:interpretation&gt;

    &lt;emma:interpretation id="better"&gt;
      &lt;emma:derived-from resource="#raw" composite="false"/&gt;
      &lt;origin&gt;Boston&lt;/origin&gt;
      &lt;destination&gt;Denver&lt;/destination&gt;
      &lt;date&gt;tomorrow&lt;/date&gt;
    &lt;/emma:interpretation&gt;
  &lt;/emma:derivation&gt;

  &lt;emma:interpretation id="best"&gt;
    &lt;emma:derived-from resource="#better" composite="false"/&gt;
    &lt;origin&gt;Boston&lt;/origin&gt;
    &lt;destination&gt;Denver&lt;/destination&gt;
    &lt;date&gt;20030315&lt;/date&gt;
  &lt;/emma:interpretation&gt;
&lt;/emma:emma&gt;
</pre>
<p>The <code>emma:derivation</code> element MAY be used as a
container for representations of the earlier stages in the
interpretation of the input. The latest stage of processing MUST be
a direct child of <code>emma:emma</code>.</p>
<p>The resource attribute on <code>emma:derived-from</code> is a
URI which can reference IDs in the current or other EMMA
documents.</p>
<p>In addition to representing sequential derivations, the EMMA
<code>emma:derived-from</code> element can also be used to capture
composite derivations. Composite derivations involve combination of
inputs from different modes.</p>
<p>In order to indicate whether an <code>emma:derived-from</code>
element describes a sequential derivation step or a composite
derivation step, the <code>emma:derived-from</code> element has an
attribute <code>composite</code> which has a boolean value. A
composite <code>emma:derived-from</code> MUST be marked as
<code>composite="true"</code> while a sequential
<code>emma:derived-from</code> element is marked as
<code>composite="false"</code>. If this attribute is not specified
the value is <code>false</code> by default.</p>
<p>In the following composite derivation example the user said
"destination" using the voice mode and circled Boston on a map
using the ink mode:</p>
<div>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:derivation&gt;
    &lt;emma:interpretation id="voice1"
        emma:start="1087995961500"
        emma:end="1087995962542"
        emma:process="http://example.com/myasr.xml"
        emma:source="http://example.com/microphone/NC-61"
        emma:signal="http://example.com/signals/sg23.wav"
        emma:confidence="0.6"
        emma:medium="acoustic"
        emma:mode="voice"
        emma:function="dialog"
        emma:verbal="true"
        emma:lang="en-US"
        emma:tokens="destination"&gt;
      &lt;rawinput&gt;destination&lt;/rawinput&gt;
    &lt;/emma:interpretation&gt;

    &lt;emma:interpretation id="ink1"
        emma:start="1087995961600"
        emma:end="1087995964000"
        emma:process="http://example.com/mygesturereco.xml"
        emma:source="http://example.com/pen/wacom123"
        emma:signal="http://example.com/signals/ink5.inkml"
        emma:confidence="0.5"
        emma:medium="tactile"
        emma:mode="ink"
        emma:function="dialog"
        emma:verbal="false"&gt;
      &lt;rawinput&gt;Boston&lt;/rawinput&gt;
    &lt;/emma:interpretation&gt;
  &lt;/emma:derivation&gt;

  &lt;emma:interpretation id="multimodal1"
      
      
      emma:confidence="0.3"
      <span>emma:start="1087995961500"</span>
      <span>emma:end="1087995964000"</span>
      emma:medium="<span>acoustic tactile</span>"
      emma:mode="<span>voice ink</span>"
      emma:function="dialog"
      emma:verbal="true"
      emma:lang="en-US"
      emma:tokens="destination"&gt;
    &lt;emma:derived-from resource="#voice1" composite="true"
    &lt;emma:derived-from resource="#ink1" composite="true"
    &lt;destination&gt;Boston&lt;/destination&gt;
  &lt;/emma:interpretation&gt;
&lt;/emma:emma&gt;
</pre></div>
<p>In this example, annotations on the multimodal interpretation
indicate the process used for the integration and there are two
<code>emma:derived-from</code> elements, one pointing to the speech
and one pointing to the pen gesture.</p>
<p>The only constraints the EMMA specification places on the
annotations that appear on a composite input are that the
<code>emma:medium</code> attribute MUST contain the union of the
<code>emma:medium</code> attributes on the combining inputs,
represented as a space delimited set of <code>nmtokens</code> as
defined in <a href="#s4.2.11">Section 4.2.11</a>, and that the
<code>emma:mode</code> attribute MUST contain the union of the
<code>emma:mode</code> attributes on the combining inputs,
represented as a space delimited set of <span><code>nmtokens</code>
as defined in <a href="#s4.2.11">Section 4.2.11</a></span>. In the
example above this meanings that the <code>emma:medium</code> value
is <code>"acoustic tactile"</code> and the <code>emma:mode</code>
attribute is <code>"voice ink"</code>. How all other annotations
are handled is author defined. In the following paragraph,
informative examples on how specific annotations might be handled
are given.</p>
<p>With reference to the illustrative example above, this paragraph
provides informative guidance regarding the determination of
annotations (beyond <code>emma:medium</code> and
<code>emma:mode</code> on a composite multimodal interpretation).
Generally the timestamp on a combined input should contain the
intervals indicated by the combining inputs. For the absolute
timestamps <code>emma:start</code> and <code>emma:end</code> this
can be achieved by taking the earlier of the
<code>emma:start</code> values
(<code>emma:start="1087995961500"</code> in our example) and the
later of the <code>emma:end</code> values
(<code>emma:end="1087995964000"</code> in the example). The
determination of relative timestamps for composite is more complex,
informative guidance is given in <a href="#s4.2.10.4">Section
4.2.10.4</a>. Generally speaking the <code>emma:confidence</code>
value will be some numerical combination of the confidence scores
assigned to the combining inputs. In our example, it is the result
of multiplying the voice and ink confidence scores
(<code>0.3</code>). In other cases there may not be a confidence
score for one of the combining inputs and the author may choose to
copy the confidence score from the input which does have one.
Generally, for <code>emma:verbal</code>, if either of the inputs
has the value <code>true</code> then the multimodal interpretation
will also be <code>emma:verbal="true"</code> as in the example. In
other words the annotation for the composite input is the result of
an inclusive OR of the boolean values of the annotations on the
inputs. If an annotation is only specified on one of the combining
inputs then it may in some cases be assumed to apply to the
multimodal interpretation of the composite input. In the example,
<code>emma:lang="en-US"</code> is only specified for the speech
input, and this annotation appears on the composite result also.
Similarly in our example, only the voice has
<code>emma:tokens</code> and the author has chosen to annotate the
combined input with the same <code>emma:tokens</code> value. In
this example, the <code>emma:function</code> is the same on both
combining input and the author has chosen to use the same
annotation on the composite interpretation.</p>
<p>In annotating derivations of the processing of the input, EMMA
provides the flexibility of both course-grained or fine-grained
annotation of relations among interpretations. For example, when
relating two N-best lists, within <code>emma:one-of</code> elements
either there can be a single <code>emma:derived-from</code> element
under <code>emma:one-of</code> referring to the ID of the
<code>emma:one-of</code> for the earlier processing stage:</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:derivation&gt;
    &lt;emma:one-of id="nbest1"
      <span>emma:medium="acoustic" emma:mode="voice"</span>&gt;
      &lt;emma:interpretation id="int1"&gt;
       &lt;res&gt;from boston to denver on march eleven two thousand three&lt;/res&gt;
      &lt;/emma:interpretation&gt;

      &lt;emma:interpretation id="int2"&gt;
       &lt;res&gt;from austin to denver on march eleven two thousand three&lt;/res&gt;
      &lt;/emma:interpretation&gt;
  &lt;/emma:one-of&gt;
&lt;/emma:derivation&gt;

&lt;emma:one-of id="nbest2"&gt;
  &lt;emma:derived-from resource="#nbest1" composite="false"/&gt;
  &lt;emma:interpretation id="int1b"&gt;
    &lt;origin&gt;Boston&lt;/origin&gt;
    &lt;destination&gt;Denver&lt;/destination&gt;
    &lt;date&gt;03112003&lt;/date&gt;
  &lt;/emma:interpretation&gt;

  &lt;emma:interpretation id="int2b"&gt;
    &lt;origin&gt;Austin&lt;/origin&gt;
    &lt;destination&gt;Denver&lt;/destination&gt;
    &lt;date&gt;03112003&lt;/date&gt;
  &lt;/emma:interpretation&gt;
&lt;/emma:one-of&gt;
  
&lt;/emma:emma&gt;
</pre>
<p>Or there can be a separate <code>emma:derived-from</code>
element on each <code>emma:interpretation</code> element referring
to the specific <code>emma:interpretation</code> element it was
derived from.</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:one-of id="nbest2"&gt;
    &lt;emma:interpretation id="int1b"&gt;
     &lt;emma:derived-from resource="#int1" composite="false"/&gt;
      &lt;origin&gt;Boston&lt;/origin&gt;
      &lt;destination&gt;Denver&lt;/destination&gt;
      &lt;date&gt;03112003&lt;/date&gt;
    &lt;/emma:interpretation&gt;

    &lt;emma:interpretation id="int2b"&gt;
     &lt;emma:derived-from resource="#int2" composite="false"/&gt;
      &lt;origin&gt;Austin&lt;/origin&gt;
      &lt;destination&gt;Denver&lt;/destination&gt;
      &lt;date&gt;03112003&lt;/date&gt;
    &lt;/emma:interpretation&gt;
  &lt;/emma:one-of&gt;
  &lt;emma:derivation&gt;
    &lt;emma:one-of id="nbest1"<br />
      <span>emma:medium="acoustic" emma:mode="voice"</span>&gt;
      &lt;emma:interpretation id="int1"&gt;
       &lt;res&gt;from boston to denver on march eleven two thousand three&lt;/res&gt;
      &lt;/emma:interpretation&gt;

      &lt;emma:interpretation id="int2"&gt;
       &lt;res&gt;from austin to denver on march eleven two thousand three&lt;/res&gt;
      &lt;/emma:interpretation&gt;
    &lt;/emma:one-of&gt;
  &lt;/emma:derivation&gt;
&lt;/emma:emma&gt;
</pre>
<p><a href="#s4.3">Section 4.3</a> provides further examples of the
use of <code>emma:derived-from</code> to represent sequential
derivations and addresses the issue of the scope of EMMA
annotations across derivations of user input.</p>
<h4 id="s4.1.3">4.1.3 Reference to grammar used:
<code>emma:grammar</code> element</h4>
<table class="defn" summary="property definition" width="98%"
cellpadding="5" cellspacing="0">
<tbody>
<tr>
<th>Annotation</th>
<th>emma:grammar</th>
</tr>
<tr>
<th>Definition</th>
<td>An element used to provide a reference to the grammar used in
processing the input.</td>
</tr>
<tr>
<th>Children</th>
<td>None</td>
</tr>
<tr>
<th>Attributes</th>
<td>
<ul>
<li><b>Required</b>:
<ul>
<li><code><span>ref</span></code> of type <code>xsd:anyURI</code>
that references a grammar used in processing the input.</li>
<li><code>id</code> of type <code>xsd:ID</code>.</li>
</ul>
</li>
</ul>
</td>
</tr>
<tr>
<th>Applies to</th>
<td>The <code>emma:grammar</code> is legal only as a child of the
<code>emma:emma</code> element.</td>
</tr>
</tbody>
</table>
<p>The grammar that was used to derive the EMMA result MAY be
specified with the <code>emma:grammar</code> annotation defined as
an element in the EMMA namespace.</p>
<p>Example:</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:grammar id="gram1" <span>ref</span>="someURI"/&gt;
  &lt;emma:grammar id="gram2" <span>ref</span>="anotherURI"/&gt;
  &lt;emma:one-of id="r1"<br />
    <span>emma:medium="acoustic" emma:mode="voice"</span>&gt;
    &lt;emma:interpretation id="int1" emma:grammar-ref="gram1"&gt;
      &lt;origin&gt;Boston&lt;/origin&gt;
    &lt;/emma:interpretation&gt;

    &lt;emma:interpretation id="int2" emma:grammar-ref="gram1"&gt;
        &lt;origin&gt;Austin&lt;/origin&gt;
    &lt;/emma:interpretation&gt;

    &lt;emma:interpretation id="int3" emma:grammar-ref="gram2"&gt;
        &lt;command&gt;help&lt;/command&gt;
    &lt;/emma:interpretation&gt;
  &lt;/emma:one-of&gt;
&lt;/emma:emma&gt;
</pre>
<p>The <code>emma:grammar</code> annotation is a child of
<code>emma:emma.</code></p>
<h3 id="s4.1.4">4.1.4 Extensibility to application/vendor specific
annotations: <code>emma:info</code> element</h3>
<table class="defn" summary="property definition" width="98%"
cellpadding="5" cellspacing="0">
<tbody>
<tr>
<th>Annotation</th>
<th>emma:info</th>
</tr>
<tr>
<th>Definition</th>
<td>The <code>emma:info</code> element acts as a container for
vendor and/or application specific metadata regarding a user's
input.</td>
</tr>
<tr>
<th>Children</th>
<td><span>One of more</span> elements in the application namespace
providing metadata about the input.</td>
</tr>
<tr>
<th>Attributes</th>
<td>
<ul>
<li><b>Optional</b>:
<ul>
<li><code>id</code> of type <code>xsd:ID</code>.</li>
</ul>
</li>
</ul>
</td>
</tr>
<tr>
<th>Applies to</th>
<td>The <code>emma:info</code> element is legal only as a child of
the EMMA elements <code>emma:emma</code>,
<code>emma:interpretation</code>, <code>emma:group</code>,
<code>emma:one-of</code>, <code>emma:sequence</code>,
<code>emma:arc</code>, or <code>emma:node</code>.</td>
</tr>
</tbody>
</table>
<p>In <a href="#s4.2">Section 4.2</a>, a series of attributes are
defined for representation of metadata about user inputs in a
standardized form. EMMA also provides an extensibility mechanism
for annotation of user inputs with vendor or application specific
metadata not covered by the standard set of EMMA annotations. The
element <code>emma:info</code> MUST be used as a container for
these annotations, UNLESS they are explicitly covered by
<code>emma:endpoint-info</code>. For example, if an input to a
dialog system needed to be annotated with the number that the call
originated from, their state, some indication of the type of
customer, and the name of the service, these pieces of information
could be represented within <code>emma:info</code> as in the
following example:</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:info&gt;
    &lt;caller_id&gt;
      &lt;phone_number&gt;2121234567&lt;/phone_number&gt;
      &lt;state&gt;NY&lt;/state&gt;
    &lt;/caller_id&gt;

    &lt;customer_type&gt;residential&lt;/customer_type&gt;
    &lt;service_name&gt;acme_travel_service&lt;/service_name&gt;
  &lt;/emma:info&gt;

  &lt;emma:one-of id="r1" emma:start="1087995961542"
      emma:end="1087995963542"
      <span>emma:medium="acoustic" emma:mode="voice"</span>&gt;
    &lt;emma:interpretation id="int1" emma:confidence="0.75"&gt;
      &lt;origin&gt;Boston&lt;/origin&gt;
      &lt;destination&gt;Denver&lt;/destination&gt;
      &lt;date&gt;03112003&lt;/date&gt;
    &lt;/emma:interpretation&gt;

    &lt;emma:interpretation id="int2" emma:confidence="0.68"&gt;
      &lt;origin&gt;Austin&lt;/origin&gt;
      &lt;destination&gt;Denver&lt;/destination&gt;
      &lt;date&gt;03112003&lt;/date&gt;
    &lt;/emma:interpretation&gt;
  &lt;/emma:one-of&gt;
&lt;/emma:emma&gt;
</pre>
<p>It is important to have an EMMA container element for
application/vendor specific annotations since EMMA elements provide
a structure for representation of multiple possible interpretations
of the input. As a result it is cumbersome to state
application/vendor specific metadata as part of the application
data within each <code>emma:interpretation</code>. An element is
used rather than an attribute so that internal structure can be
given to the annotations within <code>emma:info</code>.</p>
<p>In addition to <code>emma:emma</code>, <code>emma:info</code>
MAY also appear as a child of other structural elements such as
<code>emma:interpretation</code>, <code>emma:info</code> and so on.
When <code>emma:info</code> appears as a child of one of these
elements the application/vendor specific annotations contained
within <code>emma:info</code> are assumed to apply to all of the
<code>emma:interpretation</code> elements within the containing
element. The semantics of conflicting annotations in
<code>emma:info</code>, for example when different values are found
within <code>emma:emma</code> and <code>emma:interpretation</code>,
are left to the developer of the vendor/application specific
annotations.</p>
<h3 id="s4.1.5" class="notoc">4.1.5 Endpoint reference:
<code>emma:endpoint-info</code> element and
<code>emma:endpoint</code> element</h3>
<table class="defn" summary="property definition" width="98%"
cellpadding="5" cellspacing="0">
<tbody>
<tr>
<th>Annotation</th>
<th>emma:endpoint-info</th>
</tr>
<tr>
<th>Definition</th>
<td>The <code>emma:endpoint-info</code> element acts as a container
for all application specific annotation regarding the communication
environment.</td>
</tr>
<tr>
<th>Children</th>
<td>One or more <code>emma:endpoint</code> elements.</td>
</tr>
<tr>
<th>Attributes</th>
<td>
<ul>
<li><b>Required</b>:
<ul>
<li><code>id</code> of type <code>xsd:ID</code>.</li>
</ul>
</li>
</ul>
</td>
</tr>
<tr>
<th>Applies to</th>
<td>The <code>emma:endpoint-info</code> elements is legal only as a
child of <code>emma:emma</code>.</td>
</tr>
<tr>
<th>Annotation</th>
<th>emma:endpoint</th>
</tr>
<tr>
<th>Definition</th>
<td>The element acts as a container for application specific
endpoint information.</td>
</tr>
<tr>
<th>Children</th>
<td>Elements in the application namespace providing metadata about
the input.</td>
</tr>
<tr>
<th>Attributes</th>
<td>
<ul>
<li>Required:
<ul>
<li><code>id</code> of type <code>xsd:ID</code></li>
</ul>
</li>
<li>Optional: <code>emma:endpoint-role</code>,
<code>emma:endpoint-address</code>, <code>emma:message-id</code>,
<code>emma:port-num</code>, <code>emma:port-type</code>,
<code>emma:endpoint-pair-ref</code>,
<code>emma:service-name</code>, <code>emma:media-type</code>,
<code>emma:medium</code>, <code>emma:mode</code>.</li>
</ul>
</td>
</tr>
<tr>
<th>Applies to</th>
<td><code>emma:endpoint-info</code></td>
</tr>
</tbody>
</table>
<p>In order to conduct multimodal interaction, there is a need in
EMMA to specify the properties of the endpoint that receives the
input which leads to the EMMA annotation. This allows subsequent
components to utilize the endpoint properties as well as the
annotated inputs to conduct meaningful multimodal interaction. EMMA
element <code>emma:endpoint</code> can be used for this purpose. It
can specify the endpoint properties based on a set of common
endpoint property attributes in EMMA, such as
<code>emma:endpoint-address</code>, <code>emma:port-num</code>,
<code>emma:port-type</code>, etc. (<a href="#s4.2.14">Section
4.2.14</a>). Moreover, it provides an extensible annotation
structure that allows the inclusion of application and vendor
specific endpoint properties.</p>
<p>Note that the usage of the term "endpoint" in this context is
different from the way that the term is used in speech processing,
where it refers to the end of a speech input. As used here,
"endpoint" refers to a network location which is the source or
recipient of an EMMA document.</p>
<p>In multimodal interaction, multiple devices can be used and each
device can open multiple communication endpoints at the same time.
These endpoints are used to transmit and receive data, such as raw
input, EMMA documents, etc. The EMMA element
<code>emma:endpoint</code> provides a generic representation of
endpoint information which is relevant to multimodal interaction.
It allows the annotation to be interoperable, and it eliminates the
need for EMMA processors to create their own specialized
annotations for existing protocols, potential protocols or yet
undefined private protocols that they may use.</p>
<p>Moreover, <code>emma:endpoint-info</code> provides a container
to hold all annotations regarding the endpoint information,
including <code>emma:endpoint</code> and other application and
vendor specific annotations that are related to the communication,
allowing the same communication environment to be referenced and
used in multiple interpretations.</p>
<p>Note that EMMA provides two locations (i.e.
<code>emma:info</code> and <code>emma:endpoint-info</code>) for
specifying vendor/application specific annotations. If the
annotation is specifically related to the description of the
endpoint, then the vendor/application specific annotation SHOULD be
placed within <code>emma:endpoint-info</code>, otherwise it SHOULD
be placed within <code>emma:info</code>.</p>
<p>The following example illustrates the annotation of endpoint
reference properties in EMMA.</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"
    xmlns:ex="http://www.example.com/emma/port"&gt;
  &lt;emma:endpoint-info id="audio-channel-1"&gt;
    &lt;emma:endpoint id="endpoint1"
        emma:endpoint-role="sink"
        emma:endpoint-address="135.61.71.103"
        emma:port-num="50204"
        emma:port-type="rtp"
        emma:endpoint-pair-ref="endpoint2"
        emma:media-type="audio/dsr-202212; rate:8000; maxptime:40"
        emma:service-name="travel"
        emma:mode="voice"&gt;
      &lt;ex:app-protocol&gt;SIP&lt;/ex:app-protocol&gt;
    &lt;/emma:endpoint&gt;

    &lt;emma:endpoint id="endpoint2"
        emma:endpoint-role="source"
        emma:endpoint-address="136.62.72.104"
        emma:port-num="50204"
        emma:port-type="rtp"
        emma:endpoint-pair-ref="endpoint1"
        emma:media-type="audio/dsr-202212; rate:8000; maxptime:40"
        emma:service-name="travel"
        emma:mode="voice"&gt;
      &lt;ex:app-protocol&gt;SIP&lt;/ex:app-protocol&gt;
    &lt;/emma:endpoint&gt;
  &lt;/emma:endpoint-info&gt;

  &lt;emma:interpretation id="int1"
      emma:start="1087995961542" emma:end="1087995963542"
      emma:endpoint-info-ref="audio-channel-1"<br />
      <span>emma:medium="acoustic" emma:mode="voice"</span>&gt;
    &lt;destination&gt;Chicago&lt;/destination&gt;
  &lt;/emma:interpretation&gt;
&lt;/emma:emma&gt;
</pre>
<p>The <code>ex:app-protocol</code> is provided by the application
or the vendor specification. It specifies that the application
layer protocol used to establish the speech transmission from the
"source" port to the "sink" port is Session Initiation Protocol
(SIP). This is specific to SIP based VoIP communication, in which
the actual media transmission and the call signaling that controls
the communication sessions, are separated and typically based on
different protocols. In the above example, the Real-time
Transmission Protocol (RTP) is used in the media transmission
between the source port and the sink port.</p>
<h2 id="s4.2">4.2 EMMA annotation attributes</h2>
<h3 id="s4.2.1">4.2.1 Tokens of input: <code>emma:tokens</code>
attribute</h3>
<table class="defn" summary="property definition" width="98%"
cellpadding="5" cellspacing="0">
<tbody>
<tr>
<th>Annotation</th>
<th>emma:tokens</th>
</tr>
<tr>
<th>Definition</th>
<td>An attribute of type <code>xsd:string</code> holding a sequence
of input tokens.</td>
</tr>
<tr>
<th>Applies to</th>
<td><code>emma:interpretation</code>, <code>emma:group</code>,
<code>emma:one-of</code>, <code>emma:sequence</code>, and
application instance data.</td>
</tr>
</tbody>
</table>
<p>The <code>emma:tokens</code> annotation holds a list of input
tokens. In the following description, the term <i>tokens</i> is
used in the computational and syntactic sense of <i>units of
input</i>, and not in the sense of <i>XML tokens</i>. The value
held in <code>emma:tokens</code> is the list of the tokens of input
as produced by the processor which generated the EMMA document;
there is no language associated with this value.</p>
<p>In the case where a grammar is used to constrain input, the
value will correspond to tokens as defined by the grammar. So for
an EMMA document produced by input to a SRGS grammar [<a href=
"#SRGS">SRGS</a>], the value of <code>emma:tokens</code> will be
the list of words and/or phrases that are defined as tokens in SRGS
(<span>see</span> Section 2.1 <span>of [<a href=
"#SRGS">SRGS</a>]</span>). Items in the <code>emma:tokens</code>
list are delimited by white space and/or quotation marks for
phrases containing white space. For example:</p>
<pre class="example">
emma:tokens="arriving at 'Liverpool Street'"
</pre>
<p>where the three tokens of input are <i>arriving</i>, <i>at</i>
and <i>Liverpool Street</i>.</p>
<p>The <code>emma:tokens</code> annotation MAY be applied not just
to the lexical words and phrases of language but to any level of
input processing. Other examples of tokenization include phonemes,
ink strokes, gestures and any other discrete units of input at any
level.</p>
<p>Examples:</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:interpretation id="int1"
      emma:tokens="From Cambridge to London tomorrow"<br />
      <span>emma:medium="acoustic" emma:mode="voice"</span>&gt;
    &lt;origin emma:tokens="From Cambridge"&gt;Cambridge&lt;/origin&gt;
    &lt;destination emma:tokens="to London"&gt;London&lt;/destination&gt;
    &lt;date emma:tokens="tomorrow"&gt;20030315&lt;/date&gt;
  &lt;/emma:interpretation&gt;
&lt;/emma:emma&gt;
</pre>
<h3 id="s4.2.2">4.2.2 Reference to processing:
<code>emma:process</code> attribute</h3>
<table class="defn" summary="property definition" width="98%"
cellpadding="5" cellspacing="0">
<tbody>
<tr>
<th>Annotation</th>
<th>emma:process</th>
</tr>
<tr>
<th>Definition</th>
<td>An attribute of type <code>xsd:anyURI</code> referencing the
process used to generate the interpretation.</td>
</tr>
<tr>
<th>Applies to</th>
<td><code>emma:interpretation</code>, <code>emma:one-of</code>,
<code>emma:group</code>, <code>emma:sequence</code></td>
</tr>
</tbody>
</table>
<p>A reference to the information concerning the processing that
was used for generating an interpretation MAY be made using the
<code>emma:process</code> attribute. For example:</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:derivation&gt;
    &lt;emma:interpretation id="raw"<br />
      <span>emma:medium="acoustic" emma:mode="voice"</span>&gt;
      &lt;answer&gt;From Boston to Denver tomorrow&lt;/answer&gt;
    &lt;/emma:interpretation&gt;

    &lt;emma:interpretation id="better"
        emma:process="http://example.com/mysemproc1.xml"&gt;
      &lt;origin&gt;Boston&lt;/origin&gt;
      &lt;destination&gt;Denver&lt;/destination&gt;
      &lt;date&gt;tomorrow&lt;/date&gt;
      &lt;emma:derived-from resource="#raw"/&gt;
    &lt;/emma:interpretation&gt;
  &lt;/emma:derivation&gt;

  &lt;emma:interpretation id="best"
      emma:process="http://example.com/mysemproc2.xml"&gt;
    &lt;origin&gt;Boston&lt;/origin&gt;
    &lt;destination&gt;Denver&lt;/destination&gt;
    &lt;date&gt;03152003&lt;/date&gt;
    &lt;emma:derived-from resource="#better"/&gt;
  &lt;/emma:interpretation&gt;
&lt;/emma:emma&gt;
</pre>
<p>The process description document, referenced by the
<code>emma:process</code> annotation MAY include information on the
process itself, such as grammar, type of parser, etc. EMMA is not
normative about the format of the process description document.</p>
<h3 id="s4.2.3">4.2.3 Lack of input: <code>emma:no-input</code>
attribute</h3>
<table class="defn" summary="property definition" width="98%"
cellpadding="5" cellspacing="0">
<tbody>
<tr>
<th>Annotation</th>
<th>emma:no-input</th>
</tr>
<tr>
<th>Definition</th>
<td>Attribute holding <code>xsd:boolean</code> value that is true
if there was no input.</td>
</tr>
<tr>
<th>Applies to</th>
<td><code>emma:interpretation</code></td>
</tr>
</tbody>
</table>
<p>The case of lack of input MUST be annotated as follows:</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:interpretation id="int1" emma:no-input="true"<br />
   <span>emma:medium="acoustic" emma:mode="voice"</span>/&gt;
&lt;/emma:emma&gt;
</pre>
<p>If the <code>emma:interpretation</code> is annotated with
<code>emma:no-input="true"</code> then the
<code>emma:interpretation</code> MUST be empty.</p>
<h3 id="s4.2.4">4.2.4 Uninterpreted input:
<code>emma:uninterpreted</code> attribute</h3>
<table class="defn" summary="property definition" width="98%"
cellpadding="5" cellspacing="0">
<tbody>
<tr>
<th>Annotation</th>
<th>emma:uninterpreted</th>
</tr>
<tr>
<th>Definition</th>
<td>Attribute holding <code>xsd:boolean</code> value that is true
if <span>no interpretation was produced in response to the
input</span></td>
</tr>
<tr>
<th>Applies to</th>
<td><code>emma:interpretation</code></td>
</tr>
</tbody>
</table>
<p>An <code>emma:interpretation</code> element representing input
<span>for which no interpretation was produced</span> MUST be
annotated with <code>emma:uninterpreted="true"</code>. For
example:</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
    http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:interpretation id="interp1" emma:uninterpreted="true"<br />
   <span>emma:medium="acoustic" emma:mode="voice"</span>/&gt;
&lt;/emma:emma&gt;
</pre>
<p>The notation for uninterpreted input MAY refer to any possible
stage of interpretation processing, including raw transcriptions.
For instance, no interpretation would be produced for stages
performing pure signal capture such as audio recordings. Likewise,
if a spoken input was recognized but cannot be parsed by a language
understanding component, it can be tagged as
<code>emma:uninterpreted</code> as in the following example:</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:interpretation id="understanding"
      emma:process="http://example.com/mynlu.xml"
      emma:uninterpreted="true"
      emma:tokens="From Cambridge to London tomorrow"<br />
      <span>emma:medium="acoustic" emma:mode="voice"</span>/&gt;
&lt;/emma:emma&gt;
</pre>
<p>The <code>emma:interpretation</code> MUST be empty <span class=
"add">if</span> the <code>emma:interpretation</code> element is
annotated with <code>emma:uninterpreted="true"</code>.</p>
<h3 id="s4.2.5">4.2.5 Human language of input:
<code>emma:lang</code> attribute</h3>
<table class="defn" summary="property definition" width="98%"
cellpadding="5" cellspacing="0">
<tbody>
<tr>
<th>Annotation</th>
<th>emma:lang</th>
</tr>
<tr>
<th>Definition</th>
<td>An attribute of type <code>xsd:language</code> indicating the
language for the input.</td>
</tr>
<tr>
<th>Applies to</th>
<td><code>emma:interpretation</code>, <code>emma:group</code>,
<code>emma:one-of</code>, <code>emma:sequence</code>, and
application instance data.</td>
</tr>
</tbody>
</table>
<p>The <code>emma:lang</code> annotation is used to indicate the
human language for the input that it annotates. The values of the
<code>emma:lang</code> attribute are language identifiers as
defined by <span>IETF Best Current Practice 47 [<a href=
"#BCP47">BCP47</a>]</span>. For example,
<code>emma:lang="fr"</code> denotes French, and
<code>emma:lang="en-US"</code> denotes US English.
<code>emma:lang</code> MAY be applied to any
<code>emma:interpretation</code> element. Its annotative scope
follows the annotative scope of these elements. Unlike the
<code>xml:lang</code> attribute in XML, <code>emma:lang</code> does
not specify the language used by element contents or attribute
values.</p>
<p>The following example shows the use of <code>emma:lang</code>
for annotating an input interpretation.</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:interpretation id="int1" emma:lang="fr"<br />
    <span>emma:medium="acoustic" emma:mode="voice"</span>&gt;
    &lt;answer&gt;arretez&lt;/answer&gt;
  &lt;/emma:interpretation&gt;
&lt;/emma:emma&gt;
</pre>
<p>Many kinds of input including some inputs made through pen,
computer vision, and other kinds of sensors are inherently
non-linguistic. Examples include drawing areas, arrows etc. using a
pen and music input for tune recognition. If these non-linguistic
inputs are annotated with <code>emma:lang</code> then they MUST be
annotated as <code>emma:lang="zxx"</code>. For example, pen input
where a user circles an area on map display could be represented as
follows where <code>emma:lang="zxx"</code> indicates that the ink
input is not in any human language.</p>
<pre class="example">
<span>&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:interpretation id="pen1"
      emma:medium="tactile"
      emma:mode="ink"
      emma:lang="zxx"&gt;
    &lt;location&gt;
      &lt;type&gt;area&lt;/type&gt;
      &lt;points&gt;42.1345 -37.128 42.1346 -37.120 ... &lt;/points&gt;
    &lt;/location&gt;
  &lt;/emma:interpretation&gt;
&lt;/emma:emma&gt;</span>
</pre>
<p>If inputs for which there is no information about whether the
source input is in a particular human language, and if so which
language, are annotated with <code>emma:lang,</code> then they MUST
be annotated as <code>emma:lang=""</code>. Furthermore, in cases
where there is not explicit <code>emma:lang</code> annotation, and
none is inherited from a higher element in the document, the
default value for <code>emma:lang</code> is <code>""</code> meaning
that there is no information about whether the source input is in a
language and if so which language.</p>
<p>The <code>xml:lang</code> and <code>emma:lang</code> attributes
serve uniquely different and equally important purposes. The role
of the <code>xml:lang</code> attribute in XML 1.0 is to indicate
the language used for character data content in an XML element or
document. In contrast, the <code>emma:lang</code> attribute is used
to indicate the language employed by a user when entering an input.
Critically, <code>emma:lang</code> annotates the language of the
signal originating from the user rather than the specific tokens
used at a particular stage of processing. This is most clearly
illustrated through consideration of an example involving multiple
stages of processing of a user input. Consider the following
scenario: EMMA is being used to represent three stages in the
processing of a spoken input to an system for ordering products.
The user input is in Italian, after speech recognition, the user
input is first translated into English, then a natural language
understanding system converts the English translation into a
product ID (which is not in any particular language). Since the
input signal is a user speaking Italian, the <code>emma:lang</code>
will be <code>emma:lang="it"</code> on all of these three stages of
processing. The <code>xml:lang</code> attribute, in contrast, will
initially be <code>"it"</code>, after translation the
<code>xml:lang</code> will be <code>"en-US"</code>, and after
language understanding it will be <code>"zxx"</code> since the
product ID is non-linguistic content. The following are examples of
EMMA documents corresponding to these three processing stages,
abbreviated to show the critical attributes for discussion here.
Note that <code>&lt;transcription&gt;</code>,
<code>&lt;translation&gt;</code>, and
<code>&lt;understanding&gt;</code> are application namespace
attributes, not part of the EMMA markup.<br /></p>
<pre class="example">
<span>&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
   &lt;emma:interpretation emma:lang="it" emma:mode="voice" emma:medium="acoustic"&gt;<br />
     &lt;transcription xml:lang="it"&gt;condizionatore&lt;/transcription&gt;<br />
   &lt;/emma:interpretation&gt;
&lt;/emma:emma&gt;
</span>
</pre>
<pre class="example">
<span>&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
    http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
    &lt;emma:interpretation emma:lang="it" emma:mode="voice" emma:medium="acoustic"&gt; 
       &lt;translation xml:lang="en-US"&gt;air conditioner&lt;/translation&gt;<br />
    &lt;/emma:interpretation&gt;
&lt;/emma:emma&gt;</span>
</pre>
<pre class="example">
<span>&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
    http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
    &lt;emma:interpretation emma:lang="it" emma:mode="voice" emma:medium="acoustic"&gt; <br />
       &lt;understanding xml:lang="zxx"&gt;id1456&lt;/understanding&gt;<br />
    &lt;/emma:interpretation&gt;
&lt;/emma:emma&gt;</span>
</pre>
<p>In order <span>to</span> handle inputs involving multiple
languages, such as through code switching, the
<code>emma:lang</code> tag MAY contain several language identifiers
separated by spaces.</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:interpretation id="int1"
      emma:tokens="please stop arretez s'il vous plait"
      emma:lang="en fr"
      <span>emma:medium="acoustic" emma:mode="voice"</span>&gt;
    &lt;command&gt; CANCEL &lt;/command&gt;
  &lt;/emma:interpretation&gt;
&lt;/emma:emma&gt;
</pre>
<h3 id="s4.2.6">4.2.6 Reference to signal: <code>emma:signal</code>
<span>and <code>emma:signal-size</code></span> attributes</h3>
<table class="defn" summary="property definition" width="98%"
cellpadding="5" cellspacing="0">
<tbody>
<tr>
<th>Annotation</th>
<th>emma:signal</th>
</tr>
<tr>
<th>Definition</th>
<td>An attribute of type <code>xsd:anyURI</code> referencing the
input signal.</td>
</tr>
<tr>
<th>Applies to</th>
<td><code>emma:interpretation</code>, <code>emma:one-of</code>,
<code>emma:group</code>, <code>emma:sequence</code>,
<span>and</span> application instance data.</td>
</tr>
<tr>
<th>Annotation</th>
<th>emma:signal-size</th>
</tr>
<tr>
<th>Definition</th>
<td>An attribute <span>of type <code>xsd:nonNegativeInteger</code>
specifying</span> the size in eight bit octets of the referenced
source.</td>
</tr>
<tr>
<th>Applies to</th>
<td><code>emma:interpretation</code>, <code>emma:one-of</code>,
<code>emma:group</code>, <code>emma:sequence</code>,
<span>and</span> application instance data.</td>
</tr>
</tbody>
</table>
<p>A URI reference to the signal that originated the input
recognition process MAY be represented in EMMA using the
<code>emma:signal</code> annotation.</p>
<p>Here is an example where the reference to a speech signal is
represented using the <code>emma:signal</code> annotation on the
<code>emma:interpretation</code> element:</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:interpretation id="intp1"
      emma:signal="http://example.com/signals/sg23.bin"<br />
      <span>emma:medium="acoustic" emma:mode="voice"</span>&gt;
    &lt;origin&gt;Boston&lt;/origin&gt;
    &lt;destination&gt;Denver&lt;/destination&gt;
    &lt;date&gt;03152003&lt;/date&gt;
  &lt;/emma:interpretation&gt;
&lt;/emma:emma&gt;
</pre>
<p>The <code>emma:signal-size</code> annotation can be used to
declare the exact size of the associated signal in 8-bit octets. An
example of the use of an EMMA document to represent a recording,
with <code>emma:signal-size</code> indicating the size is as
follows:</p>
<pre class="example">
<span>
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:interpretation id="intp1"
      emma:medium="acoustic"
      emma:mode="voice"
      emma:function="recording"
      emma:uninterpreted="true"
      emma:signal="http://example.com/signals/recording.mpg"
      emma:signal-size="82102" 
      emma:duration="10000"&gt;
  &lt;/emma:interpretation&gt;
&lt;/emma:emma&gt;
</span>
</pre>
<h3 id="s4.2.7">4.2.7 Media type: <code>emma:media-type</code>
attribute</h3>
<table class="defn" summary="property definition" width="98%"
cellpadding="5" cellspacing="0">
<tbody>
<tr>
<th>Annotation</th>
<th>emma:media-type</th>
</tr>
<tr>
<th>Definition</th>
<td>An attribute of type <code>xsd:string</code> holding the MIME
type associated with the signal's data format.</td>
</tr>
<tr>
<th>Applies to</th>
<td><code>emma:interpretation</code>, <code>emma:one-of</code>,
<code>emma:group</code>, <code>emma:sequence</code>,
<code>emma:endpoint</code>, <span>and</span> application instance
data.</td>
</tr>
</tbody>
</table>
<p>The data format of the signal that originated the input MAY be
represented in EMMA using the <code>emma:media-type</code>
annotation. An initial set of MIME media types is defined by
[<a href="#RFC2046">RFC2046</a>].</p>
<p>Here is an example where the media type for the ETSI ES 202 212
audio codec for Distributed Speech Recognition (DSR) is applied to
the <code>emma:interpretation</code> element. The example also
specifies an optional sampling rate of 8 kHz and maxptime of 40
milliseconds.</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:interpretation id="intp1"<span>
        emma:signal="http://example.com/signals/signal.dsr"</span>
        emma:media-type="audio/dsr-<span>es</span>202212; rate:8000; maxptime:40"<br />
        <span>emma:medium="acoustic" emma:mode="voice"</span>&gt;
    &lt;origin&gt;Boston&lt;/origin&gt;
    &lt;destination&gt;Denver&lt;/destination&gt;
    &lt;date&gt;03152003&lt;/date&gt;
  &lt;/emma:interpretation&gt;
&lt;/emma:emma&gt;
</pre>
<h3 id="s4.2.8">4.2.8 Confidence scores:
<code>emma:confidence</code> attribute</h3>
<table class="defn" summary="property definition" width="98%"
cellpadding="5" cellspacing="0">
<tbody>
<tr>
<th>Annotation</th>
<th>emma:confidence</th>
</tr>
<tr>
<th>Definition</th>
<td>An attribute of type <code>xsd:decimal</code> in range 0.0 to
1.0, indicating the processor's confidence in the result.</td>
</tr>
<tr>
<th>Applies to</th>
<td><code>emma:interpretation</code>, <code>emma:one-of</code>,
<code>emma:group</code>, <code>emma:sequence</code>, and
application instance data.</td>
</tr>
</tbody>
</table>
<p>The confidence score in EMMA is used to indicate the quality of
the input, and if confidence is annotated on an input it MUST be
given as the value of <code>emma:confidence</code>. The confidence
score MUST be a number in the range from 0.0 to 1.0 inclusive. A
value of 0.0 indicates minimum confidence, and a value of 1.0
indicates maximum confidence. Note that
<code>emma:confidence</code> represents not only the confidence of
the speech recognizer, but rather the confidence of the whatever
processor was responsible for creating the EMMA result, based on
whatever evidence it has. For a natural language interpretation,
for example, this might include semantic heuristics in addition to
speech recognition scores. Moreover, the confidence score values do
not have to be interpreted as probabilities. In fact confidence
score values are platform-dependent, since their computation is
likely to differ between platforms and different EMMA processors.
Confidence scores are annotated explicitly in EMMA in order to
provide this information to the subsequent processes for multimodal
interaction. The example below illustrates how confidence scores
are annotated in EMMA.</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:one-of id="nbest1"<br />
    <span>emma:medium="acoustic" emma:mode="voice"</span>&gt;
    &lt;emma:interpretation id="meaning1" emma:confidence="0.6"&gt;
      &lt;location&gt;Boston&lt;/location&gt;
    &lt;/emma:interpretation&gt;

    &lt;emma:interpretation id="meaning2" emma:confidence="0.4"&gt;
      &lt;location&gt; Austin &lt;/location&gt;
    &lt;/emma:interpretation&gt;
  &lt;/emma:one-of&gt;
&lt;/emma:emma&gt;
</pre>
<p>In addition to its use as an attribute on the EMMA
interpretation and container elements, the
<code>emma:confidence</code> attribute MAY also be used to assign
confidences to elements in instance data in the application
namespace. This can be seen in the following example, where the
<code>&lt;destination&gt;</code> and <code>&lt;origin&gt;</code>
elements have confidences.</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:interpretation id="meaning1" emma:confidence="0.6"
     <span>emma:medium="acoustic" emma:mode="voice"</span>&gt;
     &lt;destination emma:confidence="0.8"&gt; Boston&lt;/destination&gt;
     &lt;origin emma:confidence="0.6"&gt; Austin &lt;/origin&gt;
  &lt;/emma:interpretation&gt;
&lt;/emma:emma&gt;
</pre>
<p>Although in general instance data can be represented in XML
using a combination of elements and attributes in the application
namespace, EMMA does not provide a standard way to annotate
processors' confidences in attributes. Consequently, instance data
that is expected to be assigned confidences SHOULD be represented
using elements, as in the above example.</p>
<h3 id="s4.2.9">4.2.9 Input source: <code>emma:source</code>
attribute</h3>
<table class="defn" summary="property definition" width="98%"
cellpadding="5" cellspacing="0">
<tbody>
<tr>
<th>Annotation</th>
<th>emma:source</th>
</tr>
<tr>
<th>Definition</th>
<td>An attribute of type <code>xsd:anyURI</code> referencing the
source of input.</td>
</tr>
<tr>
<th>Applies to</th>
<td><code>emma:interpretation</code>, <code>emma:one-of</code>,
<code>emma:group</code> , <code>emma:sequence</code>, and
application instance data.</td>
</tr>
</tbody>
</table>
<p>The source of an interpreted input MAY be represented in EMMA as
a URI resource using the <code>emma:source</code> annotation.</p>
<p>Here is an example that shows different input sources for
different input interpretations.</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"
    xmlns:myapp="http://www.example.com/myapp"&gt;
  &lt;emma:one-of id="nbest1"<br />
    <span>emma:medium="acoustic" emma:mode="voice"</span>&gt;
    &lt;emma:interpretation id="intp1"
        emma:source="http://example.com/microphone/NC-61"&gt;
      &lt;myapp:destination&gt;Boston&lt;/myapp:destination&gt;
    &lt;/emma:interpretation&gt;

    &lt;emma:interpretation id="intp2"
        emma:source="http://example.com/microphone/NC-4024"&gt;
      &lt;myapp:destination&gt;Austin&lt;/myapp:destination&gt;
    &lt;/emma:interpretation&gt;
  &lt;/emma:one-of&gt;
&lt;/emma:emma&gt;
</pre>
<h3 id="s4.2.10">4.2.10 Timestamps</h3>
<p>The start and end times for input MAY be indicated using either
absolute timestamps or relative timestamps. Both are in
milliseconds for ease in processing timestamps. Note that the
ECMAScript Date object's <code>getTime()</code> function is a
convenient way to determine the absolute time.</p>
<h4 id="s4.2.10.1">4.2.10.1 Absolute timestamps:
<code>emma:start</code>, <code>emma:end</code> attributes</h4>
<table class="defn" summary="property definition" width="98%"
cellpadding="5" cellspacing="0">
<tbody>
<tr>
<th>Annotation</th>
<th>emma:start, emma:end</th>
</tr>
<tr>
<th>Definition</th>
<td>Attributes <span>of type
<code>xsd:nonNegativeInteger</code></span> indicating the absolute
starting and ending times of an input in terms of the number of
milliseconds since 1 January 1970 00:00:00 GMT</td>
</tr>
<tr>
<th>Applies to</th>
<td><code>emma:interpretation</code>, <code>emma:group</code>,
<code>emma:one-of</code>, <code>emma:sequence</code>,
<code>emma:arc</code>, <span>and</span> application instance
data</td>
</tr>
</tbody>
</table>
<p>Here is an example of a timestamp for an absolute time.</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:interpretation id="int1"
       emma:start="1087995961542"
       emma:end="1087995963542"<br />
       <span>emma:medium="acoustic" emma:mode="voice"</span>&gt;
    &lt;destination&gt;Chicago&lt;/destination&gt;
  &lt;/emma:interpretation&gt;
&lt;/emma:emma&gt;
</pre>
<p>The <code>emma:start</code> and <code>emma:end</code>
annotations on an input MAY be identical, however the
<code>emma:end</code> value MUST NOT be less than the
<code>emma:start</code> value.</p>
<h4 id="s4.2.10.2">4.2.10.2 Relative timestamps:
<code>emma:time-ref-uri</code>,
<code>emma:time-ref-anchor-point</code>,
<code>emma:offset-to-start</code> attributes</h4>
<table class="defn" summary="property definition" width="98%"
cellpadding="5" cellspacing="0">
<tbody>
<tr>
<th>Annotation</th>
<th>emma:time-ref-uri</th>
</tr>
<tr>
<th>Definition</th>
<td>Attribute of type <code>xsd:anyURI</code> indicating the URI
used to anchor the relative timestamp.</td>
</tr>
<tr>
<th>Applies to</th>
<td><code>emma:interpretation</code>, <code>emma:group</code>,
<code>emma:one-of</code>, <code>emma:sequence</code>,
<code>emma:lattice</code>, <span>and</span> application instance
data</td>
</tr>
<tr>
<th>Annotation</th>
<th>emma:time-ref-anchor-point</th>
</tr>
<tr>
<th>Definition</th>
<td>Attribute with a value of <code>start</code> or
<code>end</code>, defaulting to <code>start</code>. It indicates
whether to measure the time from the start or end of the interval
designated with <code>emma:time-ref-uri</code>.</td>
</tr>
<tr>
<th>Applies to</th>
<td><code>emma:interpretation</code>, <code>emma:group</code>,
<code>emma:one-of</code>, <code>emma:sequence</code>,
<code>emma:lattice</code>, <span>and</span> application instance
data</td>
</tr>
<tr>
<th>Annotation</th>
<th>emma:offset-to-start</th>
</tr>
<tr>
<th>Definition</th>
<td>Attribute <span>of type <code>xsd:integer</code></span>,
defaulting to zero. It specifies the offset in milliseconds for the
start of input from the anchor point designated with
<span><code>emma:time-ref-uri</code></span> and
<span><code>emma:time-ref-anchor-point</code></span></td>
</tr>
<tr>
<th>Applies to</th>
<td><code>emma:interpretation</code>, <code>emma:group</code>,
<code>emma:one-of</code>, <code>emma:sequence</code>,
<code>emma:arc</code>, <span>and</span> application instance
data</td>
</tr>
</tbody>
</table>
<p>Relative timestamps define the start of an input relative to the
start or end of a reference interval such as another input.</p>
<p><img alt="relative timestamps" src=
"relativetimestamps.png" /></p>
<p>The reference interval is designated with
<code>emma:time-ref-uri</code> attribute. This MAY be combined with
<code>emma:time-ref-anchor-point</code> attribute to specify
whether the anchor point is the start or end of this interval. The
start of an input relative to this anchor point is then specified
with <code>emma:offset-to-start</code> attribute.</p>
<p>Here is an example where the referenced input is in the same
document:</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:sequence&gt;
    &lt;emma:interpretation id="int1"<br />
     <span>emma:medium="acoustic" emma:mode="voice"</span>&gt;
    &lt;origin&gt;Denver&lt;/origin&gt;
    &lt;/emma:interpretation&gt;
    &lt;emma:interpretation id="int2"<br />
        <span>emma:medium="acoustic" emma:mode="voice"</span>
        emma:time-ref-uri="#int1"
        emma:time-ref-anchor-point="start"
        emma:offset-to-start="5000"&gt;
    &lt;destination&gt;Chicago&lt;/destination&gt;
    &lt;/emma:interpretation&gt;
  &lt;/emma:sequence&gt;
&lt;/emma:emma&gt;
</pre>
<p>Note that the reference point refers to an input, but not
necessarily to a complete input. For example, if a speech
recognizer timestamps each word in an utterance, the anchor point
might refer to the timestamp for just one word.</p>
<p>The absolute and relative timestamps are not mutually exclusive;
that is, it is possible to have both relative and absolute
timestamp attributes on the same EMMA container element.</p>
<p>Timestamps of inputs collected by different devices will be
subject to variation if the times maintained by the devices are not
synchronized. This concern is outside of the scope of the EMMA
specification.</p>
<h4 id="s4.2.10.3">4.2.10.3 Duration of input:
<code>emma:duration</code> attribute</h4>
<table class="defn" summary="property definition" width="98%"
cellpadding="5" cellspacing="0">
<tbody>
<tr>
<th>Annotation</th>
<th>emma:duration</th>
</tr>
<tr>
<th>Definition</th>
<td>Attribute <span>of type
<code>xsd:nonNegativeInteger</code></span>, defaulting to zero. It
specifies the duration of the input in milliseconds.</td>
</tr>
<tr>
<th>Applies to</th>
<td><code>emma:interpretation</code>, <code>emma:group</code>,
<code>emma:one-of</code>, <code>emma:sequence</code>,
<code>emma:arc</code>, <span>and</span> application instance
data</td>
</tr>
</tbody>
</table>
<p>The duration of an input in milliseconds MAY be specified with
the <code>emma:duration</code> attribute. The
<code>emma:duration</code> attribute MAY be used either in
combination with timestamps or independently, for example in the
annotation of speech corpora.</p>
<p>In the following example, the duration of the signal that gave
rise to the interpretation is indicated using
<code>emma:duration</code>.</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
    &lt;emma:interpretation id="int1" emma:duration="2300"<br />
        <span>emma:medium="acoustic" emma:mode="voice"</span>&gt;
    &lt;origin&gt;Denver&lt;/origin&gt;
    &lt;/emma:interpretation&gt;
&lt;/emma:emma&gt;
</pre>
<h4 id="s4.2.10.4">4.2.10.4 Composite Input and Relative
Timestamps</h4>
<p>This section is informative.</p>
<p>The following table provides guidance on how to determine the
values of relative timestamps on a composite input.</p>
<div>
<table summary="3 columns" border="1" cellpadding="3" cellspacing=
"0">
<caption>Informative Guidance on Relative Timestamps in Composite
Derivations</caption>
<tbody>
<tr>
<td><code>emma:time-ref-uri</code></td>
<td>If the reference interval URI is the same for both inputs then
it should be the same for the composite input. If it is not the
same then relative timestamps will have to be resolved to absolute
timestamps in order to determine the combined timestamp. .</td>
</tr>
<tr>
<td><code>emma:time-ref-anchor-point</code></td>
<td>If the anchor value is the same for both inputs then it should
be the same for the composite input. If it is not the same then
relative timestamps will have to be resolved to absolute timestamps
in order to determine the combined timestamp.</td>
</tr>
<tr>
<td><code>emma:offset-to-start</code></td>
<td>Given that the <code>emma:time-ref-uri</code> and
<code>emma:time-ref-anchor-point</code> are the same for both
combining inputs, then the <code>emma:offset-to-start</code> for
the combination should be the lesser of the two. If they are not
the same then relative timestamps will have to be resolved to
absolute timestamps in order to determine the combined
timestamp.</td>
</tr>
<tr>
<td><code>emma:duration</code></td>
<td>Given that the <code>emma:time-ref-uri</code> and
<code>emma:time-ref-anchor-point</code> are the same for both
combining inputs, then the <code>emma:duration</code> is calculated
as follows. Add together the <code>emma:offset-to-start</code> and
<code>emma:duration</code> for each of the inputs. Take whichever
of these is greater and subtract from it the lesser of the
<code>emma:offset-to-start</code> values in order to determine the
combined duration. If <code>emma:time-ref-uri</code> and
<code>emma:time-ref-anchor-point</code> are not the same then
relative timestamps will have to be resolved to absolute timestamps
in order to determine the combined timestamp.</td>
</tr>
</tbody>
</table>
</div>
<h3 id="s4.2.11">4.2.11 Medium, mode, and function of user inputs:
<code>emma:medium</code>, <code>emma:mode</code>,
<code>emma:function</code>, <code>emma:verbal</code>
attributes</h3>
<table class="defn" summary="property definition" width="98%"
cellpadding="5" cellspacing="0">
<tbody>
<tr>
<th>Annotation</th>
<th>emma:medium</th>
</tr>
<tr>
<th>Definition</th>
<td>An attribute of type <span><code>xsd:nmtokens</code></span>
<span>which contains a space delimited set of values from the
set</span> {<code>acoustic</code>, <code>tactile</code>,
<code>visual</code>}.</td>
</tr>
<tr>
<th>Applies to</th>
<td><code>emma:interpretation</code>, <code>emma:group</code>,
<code>emma:one-of</code>, <code>emma:sequence</code>,
<code>emma:endpoint</code>, and application instance data</td>
</tr>
<tr>
<th>Annotation</th>
<th>emma:mode</th>
</tr>
<tr>
<th>Definition</th>
<td>An attribute of type <span><code>xsd:nmtokens</code></span>
<span>which contains a space delimited set of values from</span> an
open set of values including: {<span><code>voice</code>,
<code>dtmf</code></span>, <code>ink</code>, <code>gui</code>,
<code>keys</code>, <code>video</code>, <code>photograph</code>,
...}.</td>
</tr>
<tr>
<th>Applies to</th>
<td><code>emma:interpretation</code>, <code>emma:group</code>,
<code>emma:one-of</code>, <code>emma:sequence</code>,
<code>emma:endpoint</code>, and application instance data</td>
</tr>
<tr>
<th>Annotation</th>
<th>emma:function</th>
</tr>
<tr>
<th>Definition</th>
<td>An attribute of type <code>xsd:string</code> constrained to
values in the open set {<code>recording</code>,
<code>transcription</code>, <code>dialog</code>,
<code>verification</code>, ...}.</td>
</tr>
<tr>
<th>Applies to</th>
<td><code>emma:interpretation</code>, <code>emma:group</code>,
<code>emma:one-of</code>, <code>emma:sequence</code>, and
application instance data</td>
</tr>
<tr>
<th>Annotation</th>
<th>emma:verbal</th>
</tr>
<tr>
<th>Definition</th>
<td>An attribute of type <code>xsd:boolean</code>.</td>
</tr>
<tr>
<th>Applies to</th>
<td><code>emma:interpretation</code>, <code>emma:group</code>,
<code>emma:one-of</code>, <code>emma:sequence</code>, and
application instance data</td>
</tr>
</tbody>
</table>
<p>EMMA provides two properties for the annotation of input
modality. One indicating the broader medium or channel
(<code>emma:medium</code>) and another indicating the specific mode
of communication used on that channel (<code>emma:mode</code>). The
input medium is defined from the users perspective and indicates
whether they use their voice (<code>acoustic</code>), touch
(<code>tactile</code>), or visual appearance/motion
(<code>visual</code>) as input. Tactile includes most
<i>hand-on</i> input device types such as pen, mouse, keyboard, and
touch screen. Visual is used for camera input.</p>
<pre class="example">
emma:medium = <span>space delimited sequence of values from the set: </span>
            [acoustic|tactile|visual]
</pre>
<p>The mode property provides the ability to distinguish between
different modes of communication that may be within a particular
medium. For example, in the tactile medium, modes include
electronic ink (<code>ink</code>), and pointing and clicking on a
graphical user interface (<code>gui</code>).</p>
<pre class="example">
emma:mode = <span>space delimited sequence of values from the set: </span> 
            [<span>voice|dtmf</span>|ink|gui|keys|video|photograph| ... ]
</pre>
<p>The <code>emma:medium</code> classification is based on the
boundary between the user and the device that they use. For
<code>emma:medium="tactile"</code> the user physically touches the
device in order to provide input. For
<code>emma:medium="visual"</code> the user's movement is captured
by sensors (cameras, infrared) resulting in an input to the system.
In the case where <code>emma:medium="acoustic"</code> the user
provides input to the system by producing an acoustic signal. Note
then that DTMF input will be classified as
<code>emma:medium="tactile"</code> since in order to provide DTMF
input the user physically presses keys on a keypad.</p>
<p>While <code>emma:medium</code> and <code>emma:mode</code> are
optional on specific elements such as
<code>emma:interpretation</code> and <code>emma:one-of</code>, note
that all EMMA interpretations must be annotated for
<code>emma:medium</code> and <code>emma:mode</code>, so either
these attributes must appear directly on
<code>emma:interpretation</code> or they must appear on an ancestor
<code>emma:one-of</code> node or they must appear on an earlier
stage of the derivation listed in <code>emma:derivation</code>.</p>
<p>Orthogonal to the mode, user inputs can also be classified with
respect to their communicative function. This enables a simpler
mode classification.</p>
<pre class="example">
emma:function = [recording|transcription|dialog|verification| ... ]
</pre>
<p>For example, speech can be used for recording (e.g. voicemail),
transcription (e.g. dictation), dialog (e.g. interactive spoken
dialog systems), and verification (e.g. identifying users through
their voiceprints).</p>
<p>EMMA also supports an additional property
<code>emma:verbal</code> which distinguishes verbal use of an input
mode from non-verbal. This MAY be used to distinguish the use of
electronic ink to convey handwritten commands from the user of
electronic ink for symbolic gestures such as circles and arrows.
Handwritten commands, such as writing <i>downtown</i> in order to
change a map display to show the downtown are classified as verbal
(<code>emma:function="dialog" emma:verbal="true"</code>). Pen
gestures (arrows, lines, circles, etc), such as circling a
building, are classified as non-verbal dialog
(<code>emma:function="dialog" emma:verbal="false"</code>). The use
of handwritten words to transcribe an email message is classified
as transcription (<code>emma:function="transcription"
emma:verbal="true"</code>).</p>
<pre class="example">
emma:verbal = [true|false]
</pre>
<p>Handwritten words and ink gestures are typically recognized
using different kinds of recognition components (handwriting
recognizer vs. gesture recognizer) and the verbal annotation will
be added by the recognition component which classifies the input.
The original input source, a pen in this case, will not be aware of
this difference. The input source identifier will tell you that the
input was from a pen of some kind but will not tell you if the mode
of input was handwriting (<i>show downtown</i>) or gesture (e.g.
circling an object or area).</p>
<p>Here is an example of the EMMA annotation for a pen input where
the user's ink is recognized as either a word ("Boston") or as an
arrow:</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:one-of id="nbest1"&gt;
    &lt;emma:interpretation id="interp1"
     emma:confidence="0.6"
     emma:medium="tactile"
     emma:mode="ink"
     emma:function="dialog"
     emma:verbal="true"&gt;
       &lt;location&gt;Boston&lt;/location&gt;
    &lt;/emma:interpretation&gt;

    &lt;emma:interpretation id="interp2"
     emma:confidence="0.4"
     emma:medium="tactile"
     emma:mode="ink"
     emma:function="dialog"
     emma:verbal="false"&gt;
       &lt;direction&gt;45&lt;/direction&gt;
    &lt;/emma:interpretation&gt;
  &lt;/emma:one-of&gt;
&lt;/emma:emma&gt;
</pre>
<p>Here is an example of the EMMA annotation for a spoken command
which is recognized as either "Boston" or "Austin":</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:one-of&gt;
    &lt;emma:interpretation id="interp1"
     emma:confidence="0.6"
     emma:medium="acoustic"
     emma:mode="voice"
     emma:function="dialog"
     emma:verbal="true"&gt;
       &lt;location&gt;Boston&lt;/location&gt;
    &lt;/emma:interpretation&gt;

    &lt;emma:interpretation id="interp2"
     emma:confidence="0.4"
     emma:medium="acoustic"
     emma:mode="voice"
     emma:function="dialog"
     emma:verbal="true"&gt;
       &lt;location&gt;Austin&lt;/location&gt;
    &lt;/emma:interpretation&gt;
  &lt;/emma:one-of&gt;
&lt;/emma:emma&gt;
</pre>
<p>The following table shows the relationship between the medium,
mode, and function properties and serves as an aid for classifying
inputs. For the dialog function it also shows some examples of the
classification of inputs as verbal vs. non-verbal.</p>
<table class="modes" summary="7 columns" border="1" cellpadding="3"
cellspacing="0">
<tbody>
<tr>
<th rowspan="2">Medium</th>
<th rowspan="2">Device</th>
<th rowspan="2">Mode</th>
<th colspan="4">Function</th>
</tr>
<tr>
<th>recording</th>
<th>dialog</th>
<th>transcription</th>
<th>verification</th>
</tr>
<tr>
<td rowspan="2">acoustic</td>
<td rowspan="2">microphone</td>
<td rowspan="2">voice</td>
<td rowspan="2">audiofile (e.g. voicemail)</td>
<td>spoken command / query / response (verbal = true)</td>
<td rowspan="2">dictation</td>
<td rowspan="2">speaker recognition</td>
</tr>
<tr>
<td>singing a note (verbal = false)</td>
</tr>
<tr>
<td rowspan="14">tactile</td>
<td rowspan="2">keypad</td>
<td rowspan="2">dtmf</td>
<td rowspan="2">audiofile / character stream</td>
<td>typed command / query / response (verbal = true)</td>
<td rowspan="2">text entry (T9-tegic, word completion, or word
grammar)</td>
<td rowspan="2">password / pin entry</td>
</tr>
<tr>
<td>command key "Press 9 for sales" (verbal = false)</td>
</tr>
<tr>
<td rowspan="2">keyboard</td>
<td rowspan="2">dtmf</td>
<td rowspan="2">character / key-code stream</td>
<td>typed command / query / response (verbal = true)</td>
<td rowspan="2">typing</td>
<td rowspan="2">password / pin entry</td>
</tr>
<tr>
<td>command key "Press S for sales" (verbal = false)</td>
</tr>
<tr>
<td rowspan="4">pen</td>
<td rowspan="2">ink</td>
<td rowspan="2">trace, sketch</td>
<td>handwritten command / query / response (verbal = true)</td>
<td rowspan="2">handwritten text entry</td>
<td rowspan="2">signature, handwriting recognition</td>
</tr>
<tr>
<td>gesture (e.g. circling building) (verbal = false)</td>
</tr>
<tr>
<td rowspan="2">gui</td>
<td rowspan="2">N/A</td>
<td>tapping on named button (verbal = true)</td>
<td rowspan="2">soft keyboard</td>
<td rowspan="2">password / pin entry</td>
</tr>
<tr>
<td>drag and drop, tapping on map (verbal = false)</td>
</tr>
<tr>
<td rowspan="4">mouse</td>
<td rowspan="2">ink</td>
<td rowspan="2">trace, sketch</td>
<td>handwritten command / query / response (verbal = true)</td>
<td rowspan="2">handwritten text entry</td>
<td rowspan="2">N/A</td>
</tr>
<tr>
<td>gesture (e.g. circling building) (verbal = false)</td>
</tr>
<tr>
<td rowspan="2">gui</td>
<td rowspan="2">N/A</td>
<td>clicking named button (verbal = true)</td>
<td rowspan="2">soft keyboard</td>
<td rowspan="2">password / pin entry</td>
</tr>
<tr>
<td>drag and drop, clicking on map (verbal = false)</td>
</tr>
<tr>
<td rowspan="2">joystick</td>
<td>ink</td>
<td>trace,sketch</td>
<td>gesture (e.g. circling building) (verbal = false)</td>
<td>N/A</td>
<td>N/A</td>
</tr>
<tr>
<td>gui</td>
<td>N/A</td>
<td>pointing, clicking button / menu (verbal = false)</td>
<td>soft keyboard</td>
<td>password / pin entry</td>
</tr>
<tr>
<td rowspan="5">visual</td>
<td rowspan="2">page scanner</td>
<td rowspan="2">photograph</td>
<td rowspan="2">image</td>
<td>handwritten command / query / response (verbal = true)</td>
<td rowspan="2">optical character recognition, object/scene
recognition (markup, e.g. SVG)</td>
<td rowspan="2">N/A</td>
</tr>
<tr>
<td>drawings and images (verbal = false)</td>
</tr>
<tr>
<td>still camera</td>
<td>photograph</td>
<td>image</td>
<td>objects (verbal = false)</td>
<td>visual object/scene recognition</td>
<td>face id, retinal scan</td>
</tr>
<tr>
<td rowspan="2">video camera</td>
<td rowspan="2">video</td>
<td rowspan="2">movie</td>
<td>sign language (verbal = true)</td>
<td rowspan="2">audio/visual recognition</td>
<td rowspan="2">face id, gait id, retinal scan</td>
</tr>
<tr>
<td>face / hand / arm / body gesture (e.g. pointing, facing)
(verbal = false)</td>
</tr>
</tbody>
</table>
<h3 id="s4.2.12">4.2.12 Composite multimodality:
<code>emma:hook</code> attribute</h3>
<table class="defn" summary="property definition" width="98%"
cellpadding="5" cellspacing="0">
<tbody>
<tr>
<th>Annotation</th>
<th>emma:hook</th>
</tr>
<tr>
<th>Definition</th>
<td>An attribute of type <code>xsd:string</code> constrained to
values in the open set {<code>voice</code>, <code>dtmf</code>,
<code>ink</code>, <code>gui</code>, <code>keys</code>,
<code>video</code>, <code>photograph</code>, ...} or the wildcard
<code>any</code></td>
</tr>
<tr>
<th>Applies to</th>
<td>Application instance data</td>
</tr>
</tbody>
</table>
<p>The attribute <code>emma:hook</code> MAY be used to mark the
elements in the application semantics within an
<code>emma:interpretation</code> which are expected to be
integrated with content from input in another mode to yield a
complete interpretation. The <code>emma:mode</code> to be
integrated at that point in the application semantics is indicated
as the value of the <code>emma:hook</code> attribute. The possible
values of <code>emma:hook</code> are the list of input modes that
can be values of <code>emma:mode</code> <span>(see <a href=
"#s4.2.11">Section 4.2.11</a>)</span>. In addition to these, the
value of <code>emma:hook</code> can also be the wildcard
<code>any</code> indicating that the other content can come from
any source. The annotation <code>emma:hook</code> differs in
semantics from <code>emma:mode</code> as follows. Annotating an
element in the application semantics with
<code>emma:mode="ink"</code> indicates that that part of the
semantics came from the <code>ink</code> mode. Annotating an
element in the application semantics with
<code>emma:hook="ink"</code> indicates that part of the semantics
needs to be integrated with content from the <code>ink</code>
mode.</p>
<p>To illustrate the use of <code>emma:hook</code> consider an
example composite input in which the user says "zoom in here" in
the speech input mode while drawing an area on a graphical display
in the ink input mode. <span>The fact that the
<code>location</code> element needs to come from the
<code>ink</code> mode is indicated by annotating this application
namespace element using <code>emma:hook</code></span></p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:interpretation <span>emma:medium="acoustic"</span> emma:mode="voice"&gt;
    &lt;command&gt;
      &lt;action&gt;zoom&lt;/action&gt;
      &lt;location emma:hook="ink"&gt;
        &lt;type&gt;area&lt;/type&gt;
      &lt;/location&gt;
    &lt;/command&gt;
  &lt;/emma:interpretation&gt;
&lt;/emma:emma&gt;
</pre>
<p>For more detailed explanation of this example see <a href=
"#appC">Appendix C</a>.</p>
<h3 id="s4.2.13">4.2.13 Cost: <code>emma:cost</code> attribute</h3>
<table class="defn" summary="property definition" width="98%"
cellpadding="5" cellspacing="0">
<tbody>
<tr>
<th>Annotation</th>
<th>emma:cost</th>
</tr>
<tr>
<th>Definition</th>
<td>An attribute of type <code>xsd:decimal</code> in range 0.0 to
10000000, indicating the processor's cost or weight associated with
an input or part of an input.</td>
</tr>
<tr>
<th>Applies to</th>
<td><code>emma:interpretation</code>, <code>emma:group</code>,
<code>emma:one-of</code>, <code>emma:sequence</code>,
<code>emma:arc</code>, <code>emma:node</code>, and application
instance data.</td>
</tr>
</tbody>
</table>
<p>The cost annotation in EMMA indicates the weight or cost
associated with an user's input or part of their input. The most
common use of <code>emma:cost</code> is for representing the costs
encoded on a lattice output from speech recognition or other
recognition or understanding processes. <code>emma:cost</code> MAY
also be used to indicate the total cost associated with particular
recognition results or semantic interpretations.</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:one-of <span>emma:medium="acoustic" emma:mode="voice"</span>&gt;
    &lt;emma:interpretation id="meaning1" emma:cost="1600"&gt;
      &lt;location&gt;Boston&lt;/location&gt;
    &lt;/emma:interpretation&gt;

    &lt;emma:interpretation id="meaning2" emma:cost="400"&gt;
      &lt;location&gt; Austin &lt;/location&gt;
    &lt;/emma:interpretation&gt;
  &lt;/emma:one-of&gt;
&lt;/emma:emma&gt;
</pre>
<h3 id="s4.2.14">4.2.14 Endpoint properties:
<code>emma:endpoint-role</code>,
<code>emma:endpoint-address</code>, <code>emma:port-type</code>,
<code>emma:port-num</code>, <code>emma:message-id</code>,
<code>emma:service-name</code>, <code>emma:endpoint-pair-ref</code>,
<code>emma:endpoint-info-ref</code>
attributes</h3>
<table class="defn" summary="property definition" width="98%"
cellpadding="5" cellspacing="0">
<tbody>
<tr>
<th>Annotation</th>
<th>emma:endpoint-role</th>
</tr>
<tr>
<th>Definition</th>
<td>An attribute of type <code>xsd:string</code> constrained to
values in the set {<code>source</code>, <code>sink</code>,
<code>reply-to</code>, <code>router</code>}.</td>
</tr>
<tr>
<th>Applies to</th>
<td><code>emma:endpoint</code></td>
</tr>
<tr>
<th>Annotation</th>
<th>emma:endpoint-address</th>
</tr>
<tr>
<th>Definition</th>
<td>An attribute of type <code>xsd:anyURI</code> that uniquely
specifies the network address of the
<code>emma:endpoint</code>.</td>
</tr>
<tr>
<th>Applies to</th>
<td><code>emma:endpoint</code></td>
</tr>
<tr>
<th>Annotation</th>
<th>emma:port-type</th>
</tr>
<tr>
<th>Definition</th>
<td>An attribute of type <code>xsd:QName</code> that specifies the
type of the port.</td>
</tr>
<tr>
<th>Applies to</th>
<td><code>emma:endpoint</code></td>
</tr>
<tr>
<th>Annotation</th>
<th>emma:port-num</th>
</tr>
<tr>
<th>Definition</th>
<td>An attribute of type <code>xsd:nonNegativeInteger</code> that
specifies the port number.</td>
</tr>
<tr>
<th>Applies to</th>
<td><code>emma:endpoint</code></td>
</tr>
<tr>
<th>Annotation</th>
<th>emma:message-id</th>
</tr>
<tr>
<th>Definition</th>
<td>An attribute of type <code>xsd:anyURI</code> that specifies the
message ID associated with the data.</td>
</tr>
<tr>
<th>Applies to</th>
<td><code>emma:endpoint</code></td>
</tr>
<tr>
<th>Annotation</th>
<th>emma:service-name</th>
</tr>
<tr>
<th>Definition</th>
<td>An attribute of type <code>xsd:string</code> that specifies the
name of the service.</td>
</tr>
<tr>
<th>Applies to</th>
<td><code>emma:endpoint</code></td>
</tr>
<tr>
<th>Annotation</th>
<th>emma:endpoint-pair-ref</th>
</tr>
<tr>
<th>Definition</th>
<td>An attribute of type <code>xsd:anyURI</code> that specifies the
pairing between sink and source endpoints.</td>
</tr>
<tr>
<th>Applies to</th>
<td><code>emma:endpoint</code></td>
</tr>
<tr>
<th>Annotation</th>
<th>emma:endpoint-info-ref</th>
</tr>
<tr>
<th>Definition</th>
<td>An attribute of type <code>xsd:IDREF</code> referring to the
<code>id</code> attribute of an <code>emma:endpoint-info</code>
element.</td>
</tr>
<tr>
<th>Applies to</th>
<td><code>emma:interpretation</code>, <code>emma:group</code>,
<code>emma:one-of</code>, <code>emma:sequence</code>, and
application instance data.</td>
</tr>
</tbody>
</table>
<p>The <code>emma:endpoint-role</code> attribute specifies the role
that the particular <code>emma:endpoint</code> performs in
multimodal interaction. The role value <code>sink</code> indicates
that the particular endpoint is the receiver of the input data. The
role value <code>source</code> indicates that the particular
endpoint is the sender of the input data. The role value
<code>reply-to</code> indicates that the particular
<code>emma:endpoint</code> is the intended endpoint for the reply.
The same <code>emma:endpoint-address</code> MAY appear in multiple
<code>emma:endpoint</code> elements, provided that the same
endpoint address is used to serve multiple roles, e.g. sink,
source, reply-to, router, etc., or associated with multiple
interpretations.</p>
<p>The <code>emma:endpoint-address</code> specifies the network
address of the <code>emma:endpoint</code>, and
<code>emma:port-type</code> specifies the port type of the
<code>emma:endpoint</code>. The <code>emma:port-num</code>
annotates the port number of the endpoint (e.g. the typical port
number for an http endpoint is 80). The
<code>emma:message-id</code> annotates the message ID information
associated with the annotated input. This meta information is used
to establish and maintain the communication context for both
inbound processing and outbound operation. The service
specification of the <code>emma:endpoint</code> is annotated by
<code>emma:service-name</code> which contains the definition of the
service that the <code>emma:endpoint</code> performs. The matching
of the <code>sink</code> endpoint and its pairing
<code>source</code> endpoint is annotated by the
<code>emma:endpoint-pair-ref</code> attribute. One sink endpoint
MAY link to multiple source endpoints through
<code>emma:endpoint-pair-ref</code>. Further bounding of the
<code>emma:endpoint</code> is possible by using the annotation of
<code>emma:group</code> (see <a href="#s3.3.2">Section
3.3.2</a>).</p>
<p>The <code>emma:endpoint-info-ref</code> attribute associates the
EMMA result in the container element with an
<code>emma:endpoint-info</code> element.</p>
<p>The following example illustrates the use of these attributes in
multimodal interactions where multiple modalities are used.</p>
<pre>
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"
    xmlns:ex="http://www.example.com/emma/port"&gt;
  &lt;emma:endpoint-info id="audio-channel-1" &gt;
    &lt;emma:endpoint id="endpoint1"
        emma:endpoint-role="sink"
        emma:endpoint-address="135.61.71.103"
        emma:port-num="50204"
        emma:port-type="rtp"
        emma:endpoint-pair-ref="endpoint2"
        emma:media-type="audio/dsr-202212; rate:8000; maxptime:40"
        emma:service-name="travel"
        emma:mode="voice"&gt;
      &lt;ex:app-protocol&gt;SIP&lt;/ex:app-protocol&gt;
    &lt;/emma:endpoint&gt;

    &lt;emma:endpoint id="endpoint2" emma:endpoint-role="source"
        emma:endpoint-address="136.62.72.104"
        emma:port-num="50204"
        emma:port-type="rtp"
        emma:endpoint-pair-ref="endpoint1"
        emma:media-type="audio/dsr-202212; rate:8000; maxptime:40"
        emma:service-name="travel"
        emma:mode="voice"&gt;
      &lt;ex:app-protocol&gt;SIP&lt;/ex:app-protocol&gt;
    &lt;/emma:endpoint&gt;
  &lt;/emma:endpoint-info&gt;

  &lt;emma:endpoint-info id="ink-channel-1"&gt;
     &lt;emma:endpoint id="endpoint3" emma:endpoint-role="sink"
         emma:endpoint-address="http://emma.example/sink"
         emma:endpoint-pair-ref="endpoint4"
         emma:port-num="80" emma:port-type="http"
         emma:message-id="uuid:2e5678"
         emma:service-name="travel"
         emma:mode="ink"/&gt;
     &lt;emma:endpoint id="endpoint4"
         emma:endpoint-role="source"
         emma:port-address="http://emma.example/source"
         emma:endpoint-pair-ref="endpoint3"
         emma:port-num="80"
         emma:port-type="http"
         emma:message-id="uuid:2e5678"
         emma:service-name="travel"
         emma:mode="ink"/&gt;
  &lt;/emma:endpoint-info&gt;

  &lt;emma:group&gt;
    &lt;emma:interpretation id="int1" emma:start="1087995961542"
        emma:end="1087995963542"
        emma:endpoint-info-ref="audio-channel-1"<br />
        emma:medium="acoustic" emma:mode="voice"&gt;
      &lt;destination&gt;Chicago&lt;/destination&gt;
    &lt;/emma:interpretation&gt;

    &lt;emma:interpretation id="int2" emma:start="1087995961542"
        emma:end="1087995963542"
        emma:endpoint-info-ref="ink-channel-1"<br />
        emma:medium="acoustic" emma:mode="voice"&gt;
      &lt;location&gt;
         &lt;type&gt;area&lt;/type&gt;
         &lt;points&gt;34.13 -37.12 42.13 -37.12 ... &lt;/points&gt;
      &lt;/location&gt;
    &lt;/emma:interpretation&gt;
  &lt;/emma:group&gt;
&lt;/emma:emma&gt;
</pre>
<h3 id="s4.2.15">4.2.15 Reference to <code>emma:grammar</code>
element: <code>emma:grammar-ref</code> attribute</h3>
<table class="defn" summary="property definition" width="98%"
cellpadding="5" cellspacing="0">
<tbody>
<tr>
<th>Annotation</th>
<th>emma:grammar-ref</th>
</tr>
<tr>
<th>Definition</th>
<td>An attribute of type <code>xsd:IDREF</code> referring to the
<code>id</code> attribute of an <code>emma:grammar</code>
element<span>.</span></td>
</tr>
<tr>
<th>Applies to</th>
<td><code>emma:interpretation</code>, <code>emma:group</code>,
<code>emma:one-of</code>, <code>emma:sequence</code>.</td>
</tr>
</tbody>
</table>
<p>The <code>emma:grammar-ref</code> annotation associates the EMMA
result in the container element with an <code>emma:grammar</code>
element.</p>
<p>Example:</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:grammar id="gram1" <span>ref</span>="someURI"/&gt;

  &lt;emma:grammar id="gram2" <span>ref</span>="anotherURI"/&gt;

  &lt;emma:one-of id="r1"<br />
    <span>emma:medium="acoustic" emma:mode="voice"</span>&gt;
    &lt;emma:interpretation id="int1" emma:grammar-ref="gram1"&gt;
      &lt;origin&gt;Boston&lt;/origin&gt;
    &lt;/emma:interpretation&gt;

    &lt;emma:interpretation id="int2" emma:grammar-ref="gram1"&gt;
      &lt;origin&gt;Austin&lt;/origin&gt;
    &lt;/emma:interpretation&gt;

    &lt;emma:interpretation id="int3" emma:grammar-ref="gram2"&gt;
      &lt;command&gt;help&lt;/command&gt;
    &lt;/emma:interpretation&gt;
  &lt;/emma:one-of&gt;
&lt;/emma:emma&gt;
</pre>
<h3 id="s4.2.16">4.2.16 Reference to <code>emma:model</code>
element: <code>emma:model-ref</code> attribute</h3>
<table class="defn" summary="property definition" width="98%"
cellpadding="5" cellspacing="0">
<tbody>
<tr>
<th>Annotation</th>
<th>emma:model-ref</th>
</tr>
<tr>
<th>Definition</th>
<td>An attribute of type <code>xsd:IDREF</code> referring to the
<code>id</code> attribute of an <code>emma:model</code>
element<span>.</span></td>
</tr>
<tr>
<th>Applies to</th>
<td><code>emma:interpretation</code>, <code>emma:group</code>,
<code>emma:one-of</code>, <code>emma:sequence</code>, and
application instance data.</td>
</tr>
</tbody>
</table>
<p>The <code>emma:model-ref</code> annotation associates the EMMA
result in the container element with an <code>emma:model</code>
element.</p>
<p>Example:</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:model id="model1" ref="someURI"/&gt;

  &lt;emma:model id="model2" ref="anotherURI"/&gt;

  &lt;emma:one-of id="r1"<br />
    <span>emma:medium="acoustic" emma:mode="voice"</span>&gt;
    &lt;emma:interpretation id="int1" emma:model-ref="model1"&gt;
      &lt;origin&gt;Boston&lt;/origin&gt;
    &lt;/emma:interpretation&gt;

    &lt;emma:interpretation id="int2" emma:model-ref="model1"&gt;
      &lt;origin&gt;Austin&lt;/origin&gt;
    &lt;/emma:interpretation&gt;

    &lt;emma:interpretation id="int3" emma:model-ref="model2"&gt;
      &lt;command&gt;help&lt;/command&gt;
    &lt;/emma:interpretation&gt;
  &lt;/emma:one-of&gt;
&lt;/emma:emma&gt;
</pre>
<h3 id="s4.2.17">4.2.17 Dialog turns: <code>emma:dialog-turn</code>
attribute</h3>
<table class="defn" summary="property definition" width="98%"
cellpadding="5" cellspacing="0">
<tbody>
<tr>
<th>Annotation</th>
<th>emma:dialog-turn</th>
</tr>
<tr>
<th>Definition</th>
<td>An attribute of type <code>xsd:string</code> referring to the
dialog turn associated with a given container element.</td>
</tr>
<tr>
<th>Applies to</th>
<td><code>emma:interpretation</code>, <code>emma:group</code>,
<code>emma:one-of</code>, and <code>emma:sequence</code>.</td>
</tr>
</tbody>
</table>
<p>The <code>emma:dialog-turn</code> annotation associates the EMMA
result in the container element with a dialog turn. The syntax and
semantics of dialog turns is left open to suit the needs of
individual applications. For example, some applications might use
an integer value, where successive turns are represented by
successive integers. Other applications might combine a name of a
dialog participant with an integer value representing the turn
number for that participant. Ordering semantics for comparison of
<code>emma:dialog-turn</code> is deliberately unspecified and left
for applications to define.</p>
<p>Example:</p>
<pre class="example">
<span>
&lt;emma:emma version="1.0"
    emma="http://www.w3.org/2003/04/emma"
    xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:interpretation id="int1" emma:dialog-turn="u8"<br />
    <span>emma:medium="acoustic" emma:mode="voice"</span>&gt;
    &lt;quantity&gt;3&lt;/quantity&gt;
  &lt;/emma:interpretation&gt;
&lt;/emma:emma&gt;</span>
</pre>
<h2 class="notoc" id="s4.3">4.3 Scope of EMMA annotations</h2>
<p>The <code>emma:derived-from</code> element (<a href=
"#s4.1.2">Section 4.1.2</a>) can be used to capture both sequential
and composite derivations. This section concerns the scope of EMMA
annotations across <span>sequential</span> derivations of user
input connected using the <code>emma:derived-from</code> element
(<a href="#s4.1.2">Section 4.1.2</a>). Sequential derivations
involve processing steps that do not involve multimodal
integration, such as applying natural language understanding and
then reference resolution to a speech transcription. EMMA
derivations describe only single turns of user input and are not
intended to describe a sequence of dialog turns.</p>
<p>For example, an EMMA document could contain
<code>emma:interpretation</code> elements for the transcription,
interpretation, and reference resolution of a speech input,
utilizing the <code>id</code> values: <code>raw</code>,
<code>better</code>, and <code>best</code> respectively:</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
 &lt;emma:derivation&gt;
  &lt;emma:interpretation id="raw"
      emma:process="http://example.com/myasr1.xml"
      <span>emma:medium="acoustic" emma:mode="voice"</span>&gt;
    &lt;answer&gt;From Boston to Denver tomorrow&lt;/answer&gt;
  &lt;/emma:interpretation&gt;

  &lt;emma:interpretation id="better"
      emma:process="http://example.com/mynlu1.xml"&gt;
    &lt;emma:derived-from resource="#raw" composite="false"/&gt;
    &lt;origin&gt;Boston&lt;/origin&gt;
    &lt;destination&gt;Denver&lt;/destination&gt;
    &lt;date&gt;tomorrow&lt;/date&gt;
  &lt;/emma:interpretation&gt;
 &lt;/emma:derivation&gt;

  &lt;emma:interpretation id="best"
      emma:process="http://example.com/myrefresolution1.xml"&gt;
    &lt;emma:derived-from resource="#better" composite="false"/&gt;
    &lt;origin&gt;Boston&lt;/origin&gt;
    &lt;destination&gt;Denver&lt;/destination&gt;
    &lt;date&gt;03152003&lt;/date&gt;
  &lt;/emma:interpretation&gt;
&lt;/emma:emma&gt;
</pre>
<p>Each member of the derivation chain is linked to the previous
one by a <code>derived-from</code> element (<a href=
"#s4.1.2">Section 4.1.2</a>), which has an attribute
<code>resource</code> that provides a pointer to the
<code>emma:interpretation</code> from which it is derived. The
<code>emma:process</code> annotation (<a href="#s4.2.2">Section
4.2.2</a>) provides a pointer to the process used for each stage of
the derivation.</p>
<p>The following EMMA example represents the same derivation as
above but with a more fully specified set of annotations:</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:derivation&gt;
    &lt;emma:interpretation id="raw"
        emma:process="http://example.com/myasr1.xml"
        emma:source="http://example.com/microphone/NC-61"
        emma:signal="http://example.com/signals/sg23.wav"
        emma:confidence="0.6"
        emma:medium="acoustic"
        emma:mode="voice"
        emma:function="dialog"
        emma:verbal="true"
        emma:tokens="from boston to denver tomorrow"
        emma:lang="en-US"&gt;
      &lt;answer&gt;From Boston to Denver tomorrow&lt;/answer&gt;
    &lt;/emma:interpretation&gt;

    &lt;emma:interpretation id="better"
        emma:process="http://example.com/mynlu1.xml"
        emma:source="http://example.com/microphone/NC-61"
        emma:signal="http://example.com/signals/sg23.wav"
        emma:confidence="0.8"
        emma:medium="acoustic"
        emma:mode="voice"
        emma:function="dialog"
        emma:verbal="true"
        emma:tokens="from boston to denver tomorrow"
        emma:lang="en-US"&gt;
      &lt;emma:derived-from resource="#raw" composite="false"/&gt;
      &lt;origin&gt;Boston&lt;/origin&gt;
      &lt;destination&gt;Denver&lt;/destination&gt;
      &lt;date&gt;tomorrow&lt;/date&gt;
    &lt;/emma:interpretation&gt;
  &lt;/emma:derivation&gt;

  &lt;emma:interpretation id="best"
      emma:process="http://example.com/myrefresolution1.xml"
      emma:source="http://example.com/microphone/NC-61"
      emma:signal="http://example.com/signals/sg23.wav"
      emma:confidence="0.8"
      emma:medium="acoustic"
      emma:mode="voice"
      emma:function="dialog"
      emma:verbal="true"
      emma:tokens="from boston to denver tomorrow"
      emma:lang="en-US"&gt;
    &lt;emma:derived-from resource="#better" composite="false"/&gt;
    &lt;origin&gt;Boston&lt;/origin&gt;
    &lt;destination&gt;Denver&lt;/destination&gt;
    &lt;date&gt;03152003&lt;/date&gt;
  &lt;/emma:interpretation&gt;
&lt;/emma:emma&gt;
</pre>
<p>EMMA annotations on earlier stages of the derivation often
remain accurate at later stages of the derivation. Although this
can be captured in EMMA by repeating the annotations on each
<code>emma:interpretation</code> within the derivation, as in the
example above, there are two disadvantages of this approach to
annotation. First, the repetition of annotations makes the
resulting EMMA documents significantly more verbose. Second, EMMA
processors used for intermediate tasks such as natural language
understanding and reference resolution will need to read in all of
the annotations and write them all out again.</p>
<p>EMMA overcomes these problems by assuming that annotations on
earlier stages of a derivation automatically apply to later stages
of the derivation unless a new value is specified. Later stages of
the derivation essentially inherit annotations from earlier stages
in the derivation. For example, if there was an
<code>emma:source</code> annotation on the transcription
(<code>raw</code>) it would also apply to the later stages of the
derivation such as the result of natural language understanding
(<code>better</code>) or reference resolution
(<code>best</code>).</p>
<p>Because of the assumption in EMMA that annotations have scope
over later stages of a sequential derivation, the example EMMA
document above can be equivalently represented as follows:</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:derivation&gt;
    &lt;emma:interpretation id="raw"
        emma:process="http://example.com/myasr1.xml"
        emma:source="http://example.com/microphone/NC-61"
        emma:signal="http://example.com/signals/sg23.wav"
        emma:confidence="0.6"
        emma:medium="acoustic"
        emma:mode="voice"
        emma:function="dialog"
        emma:verbal="true"
        emma:tokens="from boston to denver tomorrow"
        emma:lang="en-US"&gt;
      &lt;answer&gt;From Boston to Denver tomorrow&lt;/answer&gt;
    &lt;/emma:interpretation&gt;

    &lt;emma:interpretation id="better"
        emma:process="http://example.com/mynlu1.xml"
        emma:confidence="0.8"&gt;
      &lt;emma:derived-from resource="#raw" composite="false"/&gt;
      &lt;origin&gt;Boston&lt;/origin&gt;
      &lt;destination&gt;Denver&lt;/destination&gt;
      &lt;date&gt;tomorrow&lt;/date&gt;
    &lt;/emma:interpretation&gt;
  &lt;/emma:derivation&gt;

  &lt;emma:interpretation id="best"
      emma:process="http://example.com/myrefresolution1.xml"&gt;
    &lt;emma:derived-from resource="#better" composite="false"/&gt;
    &lt;origin&gt;Boston&lt;/origin&gt;
    &lt;destination&gt;Denver&lt;/destination&gt;
    &lt;date&gt;03152003&lt;/date&gt;
  &lt;/emma:interpretation&gt;
&lt;/emma:emma&gt;
</pre>
<p>The fully specified derivation illustrated above is equivalent
to the reduced form derivation following it where only annotations
with new values are specified at each stage. These two EMMA
documents MUST yield the same result when processed by an EMMA
processor.</p>
<p>The <code>emma:confidence</code> annotation is respecified on
the <code>better</code> interpretation. This indicates the
confidence score for natural language understanding, whereas
<code>emma:confidence</code> on the <code>raw</code> interpretation
indicates the speech recognition confidence score.</p>
<p>In order to determine the full set of annotations that apply to
an <code>emma:interpretation</code> element an EMMA processor or
script needs to access the annotations directly on that element and
for any that are not specified follow the reference in the
<code>resource</code> attribute of the
<code>emma:derived-from</code> element to add in annotations from
earlier stages of the derivation.</p>
<p>The EMMA annotations break down into three groups with respect
to their scope in sequential derivations. One group of annotations
always hold<span>s</span> true for all members of a sequential
derivation. A second group <span>is</span> always respecified on
each stage of the derivation. A third group may or may not be
respecified.</p>
<table summary="7 columns" border="1" cellpadding="3" cellspacing=
"0">
<caption>Scope of Annotations in Sequential Derivations</caption>
<tbody>
<tr>
<th>Classification</th>
<th>Annotation</th>
</tr>
<tr>
<td rowspan="16">Applies to whole derivation</td>
<td><code>emma:signal</code></td>
</tr>
<tr>
<td><code><span>emma:signal-size</span></code></td>
</tr>
<tr>
<td><code><span>emma:dialog-turn</span></code></td>
</tr>
<tr>
<td><code>emma:source</code></td>
</tr>
<tr>
<td><code>emma:medium</code></td>
</tr>
<tr>
<td><code>emma:mode</code></td>
</tr>
<tr>
<td><code>emma:function</code></td>
</tr>
<tr>
<td><code>emma:verbal</code></td>
</tr>
<tr>
<td><code>emma:lang</code></td>
</tr>
<tr>
<td><code>emma:tokens</code></td>
</tr>
<tr>
<td><code>emma:start</code></td>
</tr>
<tr>
<td><code>emma:end</code></td>
</tr>
<tr>
<td><code>emma:time-ref-uri</code></td>
</tr>
<tr>
<td><code>emma:time-ref-anchor-point</code></td>
</tr>
<tr>
<td><code>emma:offset-to-start</code></td>
</tr>
<tr>
<td><code>emma:duration</code></td>
</tr>
<tr>
<td rowspan="2">Specified at each stage of derivation</td>
<td><code>emma:derived-from</code></td>
</tr>
<tr>
<td><code>emma:process</code></td>
</tr>
<tr>
<td rowspan="6">May be respecified</td>
<td><code>emma:confidence</code></td>
</tr>
<tr>
<td><code>emma:cost</code></td>
</tr>
<tr>
<td><code>emma:grammar-ref</code></td>
</tr>
<tr>
<td><code>emma:model-ref</code></td>
</tr>
<tr>
<td><code>emma:no-input</code></td>
</tr>
<tr>
<td><code>emma:uninterpreted</code></td>
</tr>
</tbody>
</table>
<p>One potential problem with this annotation scoping mechanism is
that earlier annotations could be lost if earlier stages of a
derivation were dropped in order to reduce message size. This
problem can be overcome by considering annotation scope at the
point where earlier derivation stages are discarded and populating
the final interpretation in the derivation with all of the
annotations which it could inherit. For example, if the
<code>raw</code> and <code>better</code> stages were dropped the
resulting EMMA document would be:</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:interpretation id="best"
      emma:start="1087995961542"
      emma:end="1087995963542"
      emma:process="http://example.com/myrefresolution1.xml"
      emma:source="http://example.com/microphone/NC-61"
      emma:signal="http://example.com/signals/sg23.wav"
      emma:confidence="0.8"
      emma:medium="acoustic"
      emma:mode="voice"
      emma:function="dialog"
      emma:verbal="true"
      emma:tokens="from boston to denver tomorrow"
      emma:lang="en-US"&gt;
    &lt;emma:derived-from resource="#better" composite="false"/&gt;
    &lt;origin&gt;Boston&lt;/origin&gt;
    &lt;destination&gt;Denver&lt;/destination&gt;
    &lt;date&gt;03152003&lt;/date&gt;
  &lt;/emma:interpretation&gt;
&lt;/emma:emma&gt;
</pre>
<p>Annotations on an <code>emma:one-of</code> element are assumed
to apply to all of the container elements within the
<code>emma:one-of</code>.</p>
<p>If <code>emma:one-of</code> appears with another
<code>emma:one-of</code> then annotations on the parent
<code>emma:one-of</code> are assumed to apply to the children of
the child <code>emma:one-of</code>.</p>
<p>Annotations on <code>emma:group</code> or
<code>emma:sequence</code> do not apply to their child
elements.</p>
<h2 id="s5">5. Conformance</h2>
<p>The contents of this section are normative.</p>
<h3 id="s5.1">5.1 Conforming EMMA Documents</h3>
<p>A document is a Conforming EMMA Document if it meets both the
following conditions:</p>
<ul>
<li>It is a well-formed XML document [<a href="#XML">XML</a>]
conforming to Namespaces in XML [<a href="#XMLNS">XMLNS</a>].</li>
<li>It adheres to the specification described in this document
(EMMA Specification) including the constraints expressed in the
Schema (see <a href="#appA">Appendix A</a>) and having an XML
Prolog and root element as specified in <a href="#s3.1">Section
3.1</a>.</li>
</ul>
<p>The EMMA specification and these conformance criteria provide no
designated size limits on any aspect of EMMA documents. There are
no maximum values on the number of elements, the amount of
character data, or the number of characters in attribute
values.</p>
<p><span>Within this specification, the term URI refers to a
Universal Resource Identifier as defined in [<a href=
"#RFC3986">RFC3986</a>] and extended in [<a href=
"#RFC3987">RFC3987</a>] with the new name IRI. The term URI has
been retained in preference to IRI to avoid introducing new names
for concepts such as "Base URI" that are defined or referenced
across the whole family of XML specifications</span>.</p>
<h3 id="s5.2">5.2 Using EMMA with other Namespaces</h3>
<p>The EMMA namespace is intended to be used with other XML
namespaces as per the Namespaces in XML Recommendation [<a href=
"#XMLNS">XMLNS</a>]. Future work by W3C is expected to address ways
to specify conformance for documents involving multiple
namespaces.</p>
<h3 id="s5.3">5.3 Conforming EMMA Processors</h3>
<p>A EMMA processor is a program that can process and/or generate
Conforming EMMA documents.</p>
<p>In a Conforming EMMA Processor, the XML parser MUST be able to
parse and process all XML constructs defined by XML 1.1 [<a href=
"#XML">XML</a>] and Namespaces in XML [<a href="#XMLNS">XMLNS</a>].
It is not required that a Conforming EMMA Processor uses a
validating XML parser.</p>
<p>A Conforming EMMA Processor MUST correctly understand and apply
the semantics of each markup element or attribute as described by
this document.</p>
<p>There is, however, no conformance requirement with respect to
performance characteristics of the EMMA Processor. For instance, no
statement is required regarding the accuracy, speed or other
characteristics of output produced by the processor. No statement
is made regarding the size of input that a EMMA Processor is
required to support.</p>
<h2 id="appendices">Appendices</h2>
<h3 id="appA">Appendix A. XML and <span>RELAX NG</span>
schemata</h3>
<p>This section is Normative.</p>
<p>This section defines the formal syntax for EMMA documents in
terms of a normative XML Schema.</p>
<p>There are both an XML Schema and <span>RELAX NG</span> Schema
for the EMMA markup. The latest version of the XML Schema for EMMA
is available at <a href=
"http://www.w3.org/TR/emma/emma.xsd">http://www.w3.org/TR/emma/emma.xsd</a>
and the RELAX NG Schema can be found at <a href=
"http://www.w3.org/TR/emma/emma.rng">http://www.w3.org/TR/emma/emma.rng</a>.</p>
<p>For stability it is RECOMMENDED that you use the dated URI
available at <a href=
"http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd">http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd</a>
and <a href=
"http://www.w3.org/TR/2009/REC-emma-20090210/emma.rng">http://www.w3.org/TR/2009/REC-emma-20090210/emma.rng</a>.</p>
<h2 id="appB">Appendix B. MIME type</h2>
<p>This section is <span>N</span>ormative.</p>
<p>This appendix registers a new MIME media type,
"<code>application/emma+xml</code>".</p>

<p>The "<code>application/emma+xml</code>" media type is
registered with IANA at
<a href="http://www.iana.org/assignments/media-types/application/">
http://www.iana.org/assignments/media-types/application/</a>.
</p>


<div>
<h3 id="media-type-registration">B.1 Registration of MIME media
type application/emma+xml</h3>
<dl>
<dt>MIME media type name:</dt>
<dd>
<p><code>application</code></p>
</dd>
<dt>MIME subtype name:</dt>
<dd>
<p><code>emma+xml</code></p>
</dd>
<dt>Required parameters:</dt>
<dd>
<p>None.</p>
</dd>
<dt>Optional parameters:</dt>
<dd>
<dl>
<dt><code>charset</code></dt>
<dd>
<p>This parameter has identical semantics to the
<code>charset</code> parameter of the <code>application/xml</code>
media type as specified in [<a href="#RFC3023">RFC3023</a>] or its
successor.</p>
</dd>
</dl>
</dd>
<dt>Encoding considerations:</dt>
<dd>
<p>By virtue of EMMA content being XML, it has the same
considerations when sent as "<code>application/emma+xml</code>"as
does XML. See RFC 3023 (or its successor), section 3.2.</p>
</dd>
<dt>Security considerations:</dt>
<dd>
<p>Several features of EMMA require dereferencing arbitrary URIs.
Implementers are advised to heed the security issues of [<a href=
"#RFC3986">RFC3986</a>] section 7.</p>
<p>In addition, because of the extensibility features for EMMA, it
is possible that "<code>application/emma+xml</code>" will describe
content that has security implications beyond those described here.
However, if the processor follows only the normative semantics of
this specification, this content will be ignored. Only in the case
where the processor recognizes and processes the additional
content, or where further processing of that content is dispatched
to other processors, would security issues potentially arise. And
in that case, they would fall outside the domain of this
registration document.</p>
</dd>
<dt>Interoperability considerations:</dt>
<dd>
<p>This specification describes processing semantics that dictate
the required behavior for dealing with, among other things,
unrecognized elements.</p>
<p>Because EMMA is extensible, conformant
"<code>application/emma+xml</code>" processors MAY expect that
content received is well-formed XML, but processors SHOULD NOT
assume that the content is valid EMMA or expect to recognize all of
the elements and attributes in the document.</p>
</dd>
<dt>Published specification:</dt>
<dd>
<p>
This media type registration is extracted from Appendix B of the
"<a href="http://www.w3.org/TR/emma/">EMMA: Extensible MultiModal Annotation markup language</a>"
specification.
</p>
</dd>
<dt>Additional information:</dt>
<dd>
<dl>
<dt>Magic number(s):</dt>
<dd>
<p>There is no single initial octet sequence that is always present
in EMMA documents.</p>
</dd>
<dt>File extension(s):</dt>
<dd>
<p>EMMA documents are most often identified with the extensions
"<code>.emma</code>"<!-- or "<code>.mma</code>"-->.</p>
</dd>
<dt>Macintosh File Type Code(s):</dt>
<dd>
<p>TEXT</p>
</dd>
</dl>
</dd>
<dt>Person &amp; email address to contact for further
information:</dt>
<dd>
<p>Kazuyuki Ashimura, &lt;<a href=
"mailto:ashimura@w3.org">ashimura@w3.org</a>&gt;.</p>
</dd>
<dt>Intended usage:</dt>
<dd>
<p>COMMON</p>
</dd>
<dt>Author/Change controller:</dt>
<dd>
<p>The EMMA specification is a work product of the World Wide Web
Consortium's Multimodal Interaction Working Group. The W3C has
change control over these specifications.</p>
</dd>
</dl>
</div>
<h2 id="appC">Appendix C. <code>emma:hook</code> and SRGS</h2>
<p>This section is <span>I</span>nformative.</p>
<div>
<p>One of the most powerful aspects of multimodal interfaces is
their ability to provide support for user inputs which are
distributed over the available input modes. These <b>composite</b>
inputs are contributions made by the user within a single turn
which have component parts in different modes. For example, the
user might say "zoom in here" in the speech mode while drawing an
area on a graphical display in the ink mode. One of the central
motivating factors for this kind of input is that different kinds
of communicative content are best suited to different input modes.
In the example of a user drawing an area on a map and saying "zoom
in here", the zoom command is easiest to provide in speech but the
spatial information, the specific area, is easier to provide in
ink.</p>
<p>Enabling composite multimodality is critical in ensuring that
multimodal systems support more natural and effective interaction
for users. In order to support composite inputs, a multimodal
architecture must provide some kind of multimodal integration
mechanism. In the W3C Multimodal Interaction Framework
<span>[<a href="#MMIF">MMI Framework</a>]</span>, multimodal
integration can be handled by an integration component which
follows the application of speech understanding and other kinds of
interpretation procedures for individual modes.</p>
<p>Given the broad range of different techniques being employed for
multimodal integration and the extent to which this is an ongoing
research problem, standardization of the specific method or
algorithm used for multimodal integration is not appropriate at
this time. In order to facilitate the development and
inter-operation of different multimodal integration mechanisms EMMA
provides markup language enabling application independent
specification of elements in the application markup where content
from another mode needs to be integrated. These representation
'hooks' can then be used by different kinds of multimodal
integration components and algorithms to drive the process of
multimodal integration. In the processing of a composite multimodal
input, the result of applying a mode-specific interpretation
component to each of the individual modes will be EMMA markup
describing the possible interpretation of that input.</p>
</div>
<p>One way to build an EMMA representation of a spoken input such
as "zoom in here" is to use grammar rules in the W3C Speech
Recognition Grammar Specification [<a href="#SRGS">SRGS</a>] using
the Semantic Interpretation <span>[<a href="#SI">SISR</a>]</span>
tags to build the application semantics with the
<code>emma:hook</code> attribute. In this approach <span>[<a href=
"#ECMASCRIPT">ECMAScript</a>]</span> is specified in order to build
up an object representing the semantics. The resulting ECMAScript
object is then translated to XML.</p>
<p>For our example case of "zoom in here". The following SRGS rule
could be used. The <span>Semantic Interpretation for Speech
Recognition</span> specification <span>[<a href=
"#SI">SISR</a>]</span> provides a reserved property
<b>_nsprefix</b> for indicating the namespace to be used with an
attribute.</p>
<pre class="example">
&lt;rule id="zoom"&gt;
  zoom in here
  &lt;tag&gt;
    $.command = new Object();
    $.command.action = "zoom";
    $.command.location = new Object();
    $.command.location._attributes = new Object();
    $.command.location._attributes.hook = new Object();
    $.command.location._attributes.hook._nsprefix = "emma";
    $.command.location._attributes.hook._value = "ink";
    $.command.location.type = "area";
  &lt;/tag&gt;
&lt;/rule&gt;
</pre>
<p>Application of this rule will result in the following ECMAScript
object being built.</p>
<pre class="example">
command: {
      action: "zoom"
      location: {
        _attributes: {
           hook: {
             _nsprefix: "emma"
             _value: "ink"
             }
           }
        type: "area"
      }
}
</pre>
<p><a href="#SI">SI</a> processing in an XML environment would
generate the following document:</p>
<pre class="example">
&lt;command&gt;
  &lt;action&gt;zoom&lt;/action&gt;
  &lt;location emma:hook="ink"&gt;
     &lt;type&gt;area&lt;/type&gt;
  &lt;/location&gt;
&lt;/command&gt;
</pre>
<p>This XML fragment might then appear within an EMMA document as
follows:</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:interpretation id="voice1"
      emma:medium="acoustic"
      emma:mode="voice"&gt;
    &lt;command&gt;
      &lt;action&gt;zoom&lt;/action&gt;
      &lt;location emma:hook="ink"&gt;
         &lt;type&gt;area&lt;/type&gt;
      &lt;/location&gt;
    &lt;/command&gt;
  &lt;/emma:interpretation&gt;
&lt;/emma:emma&gt;
</pre>
<p>The <code>emma:hook</code> annotation indicates that this speech
input needs to be combined with ink input such as the
following:</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:interpretation id="pen1"
      emma:medium="tactile"
      emma:mode="ink"&gt;
    &lt;location&gt;
      &lt;type&gt;area&lt;/type&gt;
      &lt;points&gt;42.1345 -37.128 42.1346 -37.120 ... &lt;/points&gt;
    &lt;/location&gt;
  &lt;/emma:interpretation&gt;
&lt;/emma:emma&gt;

</pre>
<p>This representation could be generated by a pen modality
component performing gesture recognition and interpretation. The
input to the component would be an <span>Ink Markup Language</span>
specification <span>[<a href="#InkML">INKML</a>]</span> of the ink
trace and the output would be the EMMA document above.</p>
<p>The combination will result in the following EMMA document for
the combined speech and pen multimodal input.</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:interpretation
      emma:medium="acoustic tactile" 
      emma:mode="<span>voice ink</span>"
      emma:process="http://example.com/myintegrator.xml"&gt;
    &lt;emma:derived-from resource="<span>http://example.com/voice1.emma/</span>#voice1" composite="true"/&gt;
    &lt;emma:derived-from resource="<span>http://example.com/pen1.emma/</span>#pen1" composite="true"/&gt;
    &lt;command&gt;
       &lt;action&gt;zoom&lt;/action&gt;
       &lt;location&gt;
         &lt;type&gt;area&lt;/type&gt;
         &lt;points&gt;42.1345 -37.128 42.1346 -37.120 ... &lt;/points&gt;
        &lt;/location&gt;
     &lt;/command&gt;
  &lt;/emma:interpretation&gt;
&lt;/emma:emma&gt;
</pre>
<div>
<p>There are two components to the process of integrating these two
pieces of semantic markup. The first is to ensure that the two are
compatible; that is, that no semantic constraints are violated. The
second is to fuse the content from the two sources. In our example,
the <code>&lt;type&gt;area&lt;/type&gt;</code> element is intended
to indicate that this speech command requires integration with an
area gesture rather than, for example, a line gesture, which would
have the subelement <code>&lt;type&gt;line&lt;/type&gt;</code>.
This constraint needs to be enforced by whatever mechanism is
responsible for multimodal integration.</p>
<p>Many different techniques could be used for achieving this
integration of the semantic interpretation of the pen input, a
<code>&lt;location&gt;</code> element, with the corresponding
<code>&lt;location&gt;</code> element in the speech. The
<span><code>emma:hook</code></span> simply serves to indicate the
existence of this relationship.</p>
<p>One way to achieve both the compatibility checking and fusion of
content from the two modes is to use a well-defined general purpose
matching mechanism such as unification. <span>Graph unification
[</span><a href="#graphunification">Graph
unification</a><span>]</span> is a mathematical operation defined
over directed acylic graphs which captures both of the components
of integration in a single operation: the applications of the
semantic constraints and the fusing of content. One possible
semantics for the <code>emma:hook</code> markup indicates that
content from the required mode needs to be unified with that
position in the application semantics. In order to unify, two
elements must not have any conflicting values for subelements or
attributes. This procedure can be defined recursively so that
elements within the subelements must also not clash and so on. The
result of unification is the union of all of the elements and
attributes of the two elements that are being unified.</p>
<p>In addition to the unification operation, in the resulting
<code>emma:interpretation</code> the <code>emma:hook</code>
attribute needs to be removed and the <code>emma:mode</code>
attribute changed to <span>the list of the modes of the individual
inputs</span> <span>, e.g. <code>"voice ink"</code></span>.</p>
<p>Instead of the unification operation, for a specific application
semantics, integration could be achieved using some other algorithm
or script. The benefit of using the unification semantics for
<code>emma:hook</code> is that it provides a general purpose
mechanism for checking the compatibility of elements and fusing
them, whatever the specific elements are in the application
specific semantic representation.</p>
<p>The benefit of using the <code>emma:hook</code> annotation for
authors is that it provides an application independent method for
indicating where integration with content from another mode is
required. If a general purpose integration mechanism is used, such
as the unification approach described above, authors should be able
to use the same integration mechanism for a range of different
applications without having to change the integration rules or
logic. For each application the speech grammar rules [<a href=
"#SRGS">SRGS</a>] need to assign <code>emma:hook</code> to the
appropriate elements in the semantic representation of the speech.
The general purpose multimodal integration mechanism will use the
<code>emma:hook</code> annotations in order to determine where to
add in content from other modes. Another benefit of the
<code>emma:hook</code> mechanism is that it facilitates
interoperability among different multimodal integration components,
so long as they are all general purpose and utilize
<code>emma:hook</code> in order to determine where to integrate
content.</p>
<p>The following provides a more detailed example of the use of the
<code>emma:hook</code> annotation. In this example, spoken input is
combined with two <span>ink</span> gestures. The semantic
representation assigned to the spoken input "send this file to
this" indicates two locations where content is required from ink
input using <code>emma:hook="ink"</code>:</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:interpretation<span> id="voice2"
      emma:medium="acoustic"
      emma:mode="voice"
      emma:tokens="send this file to this"
      emma:start="1087995961500"
      emma:end="1087995963542"</span>&gt;
    &lt;command&gt;
      &lt;action&gt;send&lt;/action&gt;
        &lt;arg1&gt;
          &lt;object emma:hook="ink"&gt;
            &lt;type&gt;file&lt;/type&gt;
            &lt;number&gt;1&lt;/number&gt;
          &lt;/object&gt;
        &lt;/arg1&gt;
       &lt;arg2&gt;
         &lt;object emma:hook="ink"&gt;
           &lt;number&gt;1&lt;/number&gt;
         &lt;/object&gt;
       &lt;/arg2&gt;
    &lt;/command&gt;
  &lt;/emma:interpretation&gt;
&lt;/emma:emma&gt;
</pre>
<p>The user gesturing on the two locations on the display can be
represented using <code>emma:sequence</code>:</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
  &lt;emma:sequence<span> id="ink2"</span>&gt;
    &lt;emma:interpretation <span>emma:start="1087995960500"
      emma:end="1087995960900"<br />
      emma:medium="tactile"
      emma:mode="ink"</span>&gt;
      &lt;object&gt;
       &lt;type&gt;file&lt;/type&gt;
       &lt;number&gt;1&lt;/number&gt;
       &lt;id&gt;test.pdf&lt;/id&gt;
      &lt;object&gt;
    &lt;/emma:interpretation&gt;
    &lt;emma:interpretation <span>emma:start="1087995961000"
      emma:end="1087995961100"<br />
      emma:medium="tactile"
      emma:mode="ink"</span>&gt;
      &lt;object&gt;
        &lt;type&gt;printer&lt;/type&gt;
        &lt;number&gt;1&lt;/number&gt;
        &lt;id&gt;lpt1&lt;/id&gt;
      &lt;object&gt;
    &lt;/emma:interpretation&gt;
  &lt;/emma:sequence&gt;
&lt;/emma:emma&gt;
</pre>
<p>A general purpose unification-based multimodal integration
algorithm could use the <code>emma:hook</code> annotation as
follows. It identifies the elements marked with
<code>emma:hook</code> in document order. For each of those in
turn, it attempts to unify the element with the corresponding
element in order in the <code>emma:sequence</code>. Since none of
the subelements conflict, the unification goes through and as a
result, we have the following EMMA for the composite result:</p>
<pre class="example">
&lt;emma:emma version="1.0"
    xmlns:emma="http://www.w3.org/2003/04/emma"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.w3.org/2003/04/emma
     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"
    xmlns="http://www.example.com/example"&gt;
&lt;emma:interpretation<span> id="multimodal2"
      emma:medium="acoustic tactile"
      emma:mode="voice ink"
      emma:tokens="send this file to this"
      emma:process="http://example.com/myintegration.xml"
      emma:start="1087995960500"
      emma:end="1087995963542"</span>&gt;
  &lt;emma:derived-from resource="<span>http://example.com/voice2.emma/</span>#voice2" composite="true"/&gt;
  &lt;emma:derived-from resource="<span>http://example.com/ink2.emma/</span>#ink2" composite="true"/&gt;
  &lt;command&gt;
   &lt;action&gt;send&lt;/action&gt;
    &lt;arg1&gt;
     &lt;object&gt;
       &lt;type&gt;file&lt;/type&gt;
       &lt;number&gt;1&lt;/number&gt;
        &lt;id&gt;test.pdf&lt;/id&gt;
     &lt;/object&gt;
    &lt;/arg1&gt;
    &lt;arg2&gt;
     &lt;object&gt;
       &lt;type&gt;printer&lt;/type&gt;
        &lt;number&gt;1&lt;/number&gt;
       &lt;id&gt;lpt1&lt;/id&gt;
     &lt;/object&gt;
    &lt;/arg2&gt;
  &lt;/command&gt;
&lt;/emma:interpretation&gt;
&lt;/emma:emma&gt;
</pre></div>
<h2 id="appD">Appendix D. EMMA event interface</h2>
<p>This section is <span>I</span>nformative.</p>
<p>The W3C Document Object Model [<a href="#DOM">DOM</a>] defines
platform and language neutral interfaces that gives programs and
scripts the means to dynamically access and update the content,
structure and style of documents. DOM Events define a generic event
system which allows registration of event handlers, describes event
flow through a tree structure, and provides basic contextual
information for each event.</p>
<p>This section of the EMMA specification extends the DOM Event
interface for use with events that describe interpreted user input
in terms of a DOM Node for an EMMA document.</p>
<pre class="example">
// File: emma.idl

#ifndef _EMMA_IDL_
#define _EMMA_IDL_

#include "dom.idl"#include "views.idl"#include "events.idl"
#pragma prefix "dom.w3c.org"module emma
{
  typedef dom::DOMString DOMString;
  typedef dom::Node Node;

  interface EMMAEvent : events::UIEvent {
    readonly attribute dom::Node  node;
    void               initEMMAEvent(in DOMString typeArg,
                                   in boolean canBubbleArg,
                                   in boolean cancelableArg,
                                   in Node node);
  };
};

#endif // _EMMA_IDL_
</pre>
<h2 id="appE">Appendix E. References</h2>
<h3 id="appE1">E.1 Normative references</h3>
<dl>
<dt id="BCP47">BCP47</dt>
<dd>A. Phillips and M. Davis, editors. <a href=
"http://www.rfc-editor.org/rfc/bcp/bcp47.txt">Tags for the
Identification of Languages</a>, IETF, September 2006.</dd>
<dt id="RFC3023">RFC3023</dt>
<dd>M. Murata et al.<span>,</span> editors. <a href=
"http://www.ietf.org/rfc/rfc3023.txt">XML Media Types</a>. IETF RFC
3023<span>, January 2001</span>.</dd>
<dt id="RFC2046">RFC2046</dt>
<dd>N. Freed and N. Borenstein<span>,</span> editors. <a href=
"http://www.ietf.org/rfc/rfc2046.txt">Multipurpose Internet Mail
Extensions (MIME) Part Two: Media Types</a>. IETF RFC 2046<span>,
November 1996</span>.</dd>
<dt><a id="ref-rfc2119" name="ref-rfc2119" shape=
"rect">RFC2119</a></dt>
<dd>S. Bradner, <span>e</span>ditor. <a href=
"http://www.ietf.org/rfc/rfc2119.txt">Key words for use in RFCs to
Indicate Requirement Levels</a>, IETF <span>RFC 2119</span>, March
1997.</dd>
<dt id="RFC3986">RFC3986</dt>
<dd>T. Berners-Lee et al.<span>,</span> editors. <a href=
"http://www.ietf.org/rfc/rfc3986.txt">Uniform Resource Identifier
(URI): Generic Syntax</a>. IETF RFC 3986<span>, January
2005</span>.</dd>
<dt id="RFC3987">RFC3987</dt>
<dd>M. Duerst and M. Suignard<span>,</span> editors. <a href=
"http://www.ietf.org/rfc/rfc3987.txt">Internationalized Resource
Identifiers (IRIs)</a>. IETF RFC 3987<span>, January
2005</span>.</dd>
<dt id="XML">XML</dt>
<dd>Tim Bray <span>et al.,</span> editors. <a href=
"http://www.w3.org/TR/2004/REC-xml11-20040204/">Extensible Markup
Language (XML) 1.1</a>. World Wide Web Consortium, <span>W3C
Recommendation,</span> 2004.</dd>
<dt id="XMLNS">XMLNS</dt>
<dd>Tim Bray <span>et al.</span>, editors<span>.</span> <a href=
"http://www.w3.org/TR/xml-names11/">Namespaces in XML 1.1</a>,
World Wide Web Consortium, <span>W3C Recommendation,</span>
200<span>6</span>.</dd>
<dt id="XSD1">XML Schema Structures</dt>
<dd>Henry S. Thompson <span>et al.</span>, editors. <a href=
"http://www.w3.org/TR/xmlschema-1/">XML Schema Part 1: Structures
Second Edition</a>, World Wide Web Consortium<span>, W3C
Recommendation</span>, 2004.</dd>
<dt id="XSD2">XML Schema Datatypes</dt>
<dd>Paul V. Biron <span>and</span> Ashok Malhotra, editors.
<a href="http://www.w3.org/TR/xmlschema-2/">XML Schema Part 2:
Datatypes Second Edition</a>, World Wide Web Consortium, <span>W3C
Recommendation,</span> 2004.</dd>
</dl>
<h3 id="appE2">E.2 Informative references</h3>
<dl>
<dt id="DOM">DOM</dt>
<dd><a href="http://www.w3.org/DOM/">Document Object Model</a>,
World Wide Web Consortium, 2005.</dd>
<dt id="ECMASCRIPT">ECMAScript</dt>
<dd><a href=
"http://www.ecma-international.org/publications/files/ECMA-ST/Ecma-262.pdf">
ECMAScript</a></dd>
<dt id="InkML">INKML</dt>
<dd>Yi-Min Chee, Max Froumentin, Stephen M. Watt, editors. <a href=
"http://www.w3.org/TR/InkML/">Ink Markup Language (InkML)</a>,
World Wide Web Consortium, W3C Working Draft, 2006.</dd>
<dt id="SI">SI<span>SR</span></dt>
<dd>Luc Van Tichelen <span>and Dave Burke</span>,
editor<span>s</span>. <a href=
"http://www.w3.org/TR/semantic-interpretation/">Semantic
Interpretation for Speech Recognition</a>, World Wide Web
Consortium, <span>W3C Proposed Recommendation, 2007</span>.</dd>
<dt id="SRGS">SRGS</dt>
<dd>Andrew Hunt, Scott McGlashan, editors. <a href=
"http://www.w3.org/TR/speech-grammar/">Speech Recognition Grammar
Specification Version 1.0</a>, World Wide Web Consortium<span>, W3C
Recommendation,</span> 2004.</dd>
<dt id="XFORMS">XFORMS</dt>
<dd><span>John M. Boyer et al., editors.</span> <a href=
"http://www.w3.org/TR/2006/REC-xforms-20060314/">XForms <span>1.0
(Second Edition)</span></a>, World Wide Web Consortium, <span>W3C
Recommendation,</span> 2006.</dd>
<dt id="RELAXNG">RELAX-NG</dt>
<dd><span>James Clark and Makoto Murata, editors.</span> <a href=
"http://www.oasis-open.org/committees/relax-ng/spec-20011203.html"><span>
RELAX NG Specification</span></a><span>, OASIS, Committee
Specification, 2001.</span></dd>
<dt id="EMMAreqs">EMMA Requirements</dt>
<dd>Stephane H. Maes and Stephen Potter, editors. <a href=
"http://www.w3.org/TR/EMMAreqs/">Requirements for EMMA</a>, World
Wide Web Consortium, <span>W3C Note,</span> 2003<span>.</span></dd>
<dt id="graphunification">Graph Unification</dt>
<dd>Bob Carpenter. <cite>The Logic of Typed Feature
Structures</cite>, Cambridge Tracts in Theoretical Computer Science
32, Cambridge University Press, 1992.</dd>
<dd>Kevin Knight. <cite>Unification: A Multidisciplinary
Survey</cite>, ACM Computing Surveys, 21(1), 1989.</dd>
<dd>Michael Johnston. <cite>Unification-based Multimodal
Parsing</cite>, Proceedings of Association for Computational
Linguistics, pp. 624-630, 1998.</dd>
<dt id="MMIF">MMI Framework</dt>
<dd>James A. Larson, T.V. Raman and Dave Raggett, editors. <a href=
"http://www.w3.org/TR/mmi-framework/">W3C Multimodal Interaction
Framework</a>, World Wide Web Consortium<span>, W3C Note</span>,
2003<span>.</span></dd>
<dt id="MMIreqs">MMI Requirements</dt>
<dd>Stephane H. Maes and Vijay Saraswat, editors. <a href=
"http://www.w3.org/TR/mmi-reqs/">Multimodal Interaction
Requirements</a>, World Wide Web Consortium<span>, W3C Note</span>,
2003<span>.</span></dd>
</dl>
<h2 id="appF">Appendix F. Changes since last draft</h2>
<p>This section is <span>I</span>nformative.</p>
<p>
Since the publication of the Proposed Recommendation of the EMMA
specification, the following minor editorial changes have been
added to the draft.
</p>
<ul>
<li>
Fixed wrong style of text.
(<a href="#s1.2">1.2 Terminology</a>)
</li>

<li>
Changed schemaLocation URI in example codes
  from
  "http://www.w3.org/TR/2008/PR-emma-20081215/"
  to 
  "http://www.w3.org/TR/2009/REC-emma-20090210/".
(<a href="#s2">2. Structure of EMMA documents</a>,
<a href="#s3">3. EMMA structural elements</a>
and
<a href="#s4">4 EMMA annotations</a>)
</li>

<li>
Changed the note on the status of MIME type registration from
  "being submitted to the IESG for review, approval, and registration
  with IANA" to "registered with IANA at
  http://www.iana.org/assignments/media-types/application/" because
  the EMMA MIME type is registered with IANA.
(<a href="#appB">Appendix B</a>)
</li>
</ul>

<h2 id="appG">Appendix G. Acknowledgements</h2>
<p>This section is <span>I</span>nformative.</p>
<p>The editors would like to recognize the contributions of the
current and former members of the W3C Multimodal Interaction Group
<em>(listed in alphabetical order)</em>:</p>
<dl>
<dd>Kazuyuki Ashimura, W3C</dd>
<dd>Patrizio Bergallo, (until 2008, while at Loquendo)</dd>
<dd>Wu Chou, Avaya</dd>
<dd>Max Froumentin, (until 2006, while at W3C)</dd>
<dd>Katriina Halonen, Nokia</dd>
<dd>Jin Liu, T-Systems</dd>
<dd>Roberto Pieraccini, Speechcycle</dd>
<dd>Stephen Potter, Microsoft</dd>
<dd>Massimo Romanelli, DFKI</dd>
<dd>Yuan Shao, Canon</dd>
</dl>
</body>
</html>