IAAI-14: Review Forms

The Twenty-Sixth Annual Conference on Innovative Applications of Artificial Intelligence

Collocated with the Twenty-Eighth Conference on Artificial Intelligence
Quebec City, Quebec, Canada July 29–31, 2014

IAAI-14 Review Forms

Deployed Applications Paper Review Form

We suggest the following format: (1) Summarize the content of the article; (2) Discuss its contribution; (3) Discuss its flaws; (4) Provide suggestions for improvement.

Is the paper a Deployed Application? [YES or NO]

Does the paper describe a deployed application that has been used regularly for at least several months? (The period of regular use can include in-house testing prior to deployment.)

[If you answered NO, the paper is not suitable for the "deployed applications" track. Please review it for possible inclusion in the "emerging technology" track.]

Significance: How important is the problem being addressed? Is it a difficult or simple problem? Is it central or peripheral to a category of applications? Is the tool or methodology presented generally applicable or domain specific? Does the tool or methodology offer the potential for new or more powerful applications of AI?

AI Technology: Does the paper identify AI research needed for a particular application or class of applications? Does the paper characterize the needs of application domains for solutions of particular AI problems? Does the paper evaluate the applicability of an AI tool or methodology for an application domain? Does the paper describe AI technology that could enable new or more powerful AI applications?

Innovation: Does the tool, technique, or method advance the state-of-the-art or state-of-the-practice of AI technology? Does the tool, technique, or method address a new or previously reported problem? If it is a previously reported problem, does the tool, technique, or method solve it in a different, new, more effective, or more efficient way? Does the reported work integrate AI with other AI or non-AI technologies in a new way? Does the work provide a new perspective on an application domain? Does the work apply AI to a new domain?

Content: Does the paper motivate the need for the tool or methodology? Does the paper adequately describe the task it performs or the problem it solves? Does it provide technical details about the design and implementation of the tool or methodology? Does the paper clearly identify the AI research results on which the tool or methodology depends? Does it relate the tool or methodology to the needs of application domains? Does it provide insights about the use of AI technology in general or for a particular application domain? Does it describe the development process and costs? Does it discuss estimated or measured benefits? Does it detail the evaluation method and results?

Evaluation: Has the tool or methodology been tested on real data? Has it been evaluated by end users? Has it been incorporated into a deployed application? Does the application produce measurable benefits? Has it been compared to other competing tools or methods? Have the results been published in the application literature?

Technical Quality: Is the paper technically sound? Does it carefully evaluate the strengths and limitations of its contribution? Are the results described and evaluated? Are its claims backed up? Does it identify and describe relevant previous work?

Clarity: Is the paper clearly written? Is it organized logically? Are there sufficient figures and examples to illustrate the key points? Is the paper accessible to those outside the application domain? Is it accessible to those in other technical specialties?

Task or Problem Description: Describe the task the application performs or the problem it solves. State the objectives of the application and explain why an AI solution was important. If other solutions were tried and failed outline these solutions and the reasons for their failure.

Application Description: Describe the application, providing key technical details about design and implementation. What are the system components, what are their functions, and how do they interact? What languages and tools are used in the application? How is knowledge represented? What is the hardware and software environment in which the system is deployed? Provide examples to illustrate how the system is used.

Uses of AI Technology: On what AI research results does the application depend? What key aspects of AI technology allowed the application to succeed? How were the techniques modified to fit the needs of the application? If applicable, describe how AI technology is integrated with other technology. If a commercial tool is used, explain the decision criteria used to select it. Describe any insights gained about the application of AI technology. What AI approaches or techniques were tried and did not work? Why not?

Application Use and Payoff: How long has this application been deployed? Explain how widely, how often, and by whom the application is being used. Also describe the application's payoff. What measurable benefits have resulted from its use? What additional benefits do you expect over time? What impacts has it had on the users' business processes?

Application Development and Deployment: Describe the development and deployment process. How long did they take? How many developers were involved? What were the costs? What were the difficulties, and how were they overcome? What are the lessons learned? What, if any, formal development methods were used?

Maintenance: Describe your experience with and plans for maintenance of the application. Who maintains the application? How often is update needed? Is domain knowledge expected to change over time? How does the design of the application facilitate update?

Other Comments:

Emerging Technology Paper Review Form

We suggest the following format: (1) Summarize the content of the article; (2) Discuss its contribution; (3) Discuss its flaws; (4) Provide suggestions for improvement.

Significance: How important is the problem being addressed? Is it a difficult or simple problem? Is it central or peripheral to a category of applications? Is the tool or methodology presented generally applicable or domain specific? Does the tool or methodology offer the potential for new or more powerful applications of AI?

AI Technology: Does the paper identify AI research needed for a particular application or class of applications? Does the paper characterize the needs of application domains for solutions of particular AI problems? Does the paper evaluate the applicability of an AI tool or methodology for an application domain? Does the paper describe AI technology that could enable new or more powerful AI applications?

Innovation: Does the tool, technique, or method advance the state-of-the-art or state-of-the-practice of AI technology? Does the tool, technique, or method address a new or previously reported problem? If it is a previously reported problem, does the tool, technique, or method solve it in a different, new, more effective, or more efficient way? Does the reported work integrate AI with other AI or non-AI technologies in a new way? Does the work provide a new perspective on an application domain? Does the work apply AI to a new domain?

Content: Does the paper motivate the need for the tool or methodology? Does the paper adequately describe the task it performs or the problem it solves? Does it provide technical details about the design and implementation of the tool or methodology? Does the paper clearly identify the AI research results on which the tool or methodology depends? Does it relate the tool or methodology to the needs of application domains? Does it provide insights about the use of AI technology in general or for a particular application domain? Does it describe the development process and costs? Does it discuss estimated or measured benefits? Does it detail the evaluation method and results?

Evaluation (Emerging Papers): Has the tool or methodology been tested on real data? Has it been evaluated by end users? Has it been compared to other competing tools or methods? Does the work demonstrate the possible benefits of application? Have the results been published in the application literature?

Technical Quality: Is the paper technically sound? Does it carefully evaluate the strengths and limitations of its contribution? Are the results described and evaluated? Are its claims backed up? Does it identify and describe relevant previous work?

Clarity: Is the paper clearly written? Is it organized logically? Are there sufficient figures and examples to illustrate the key points? Is the paper accessible to those outside the application domain? Is it accessible to those in other technical specialties?

Challenge Paper Review Form

We suggest the following format: (1) Summarize the content of the article; (2) Discuss its contribution; (3) Discuss its flaws; (4) Provide suggestions for improvement.

Significance: How important is the proposed challenge problem? Is it a difficult or simple problem? Is it central or peripheral to a category of applications?

Innovation: Would a solution to the challenge problem advance the state of the art or state of the practice of AI technology? Does the challenge problem require an integration of AI with other AI or non-AI technologies in a new way? Are solutions to the problem likely to provide a new perspective on an application domain? Does problem require the application of AI to a new domain?

AI Technology: Does the paper characterize the AI research needed for a particular application or class of applications? Does the paper characterize the needs of application domains for solutions of particular AI problems?

Content: Does the paper motivate the challenge problem? Does the paper adequately describe the task or problem? Does it discuss the needs of the application domain? Does it provide insights about the potential use of AI technology in general or for a particular application domain?

Evaluation: Does the paper indicate a source for data relevant to the problem? Does the paper indicate how researchers might artificially generate data for the problem? Does the paper provide evaluation criteria for proposed solutions?

Clarity: Is the paper clearly written? Is the paper accessible to those outside the application domain? Is it accessible to those in other technical specialties?

Links