Objectives:

• Evaluate your,
  • manual testing skills
  • product evaluation skills
  • effort estimation skills
• Peer-evaluate your
  • product design
  • implementation effort
  • documentation quality

When, where: Week 13 lecture

Grading:

• Your performance in the practical exams will be considered for your final grade (under the QA category and under Implementation category, about 10 marks in total).
• You will be graded based on your effectiveness as a tester (e.g., the percentage of the bugs you found, the nature of the bugs you found) and how far off your evaluation/estimates are from the evaluator consensus.  Explanation: we understand that you have limited expertise in this area; hence, we penalize only if your inputs don't seem to be based on a sincere effort to test/evaluate.
• The bugs found in your product by others will affect your v1.4 marks. You will be given a chance to reject false-positive bug reports.

During:

1. Take note of your team to test. It will be given to you by the teaching team (distributed via IVLE gradebook).
2. Download from IVLE all files submitted by the team (i.e. jar file, User Guide, Developer Guide, and Project Portfolio Pages) into an empty folder.
3. [~45 minutes] Test the product and report bugs as described below:

1. [~45 minutes] Evaluate the following aspects. Note down your evaluation in a hard copy (as a backup). Submit via TEAMMATES.

   • A. Cohesiveness of product features []: Do the features fit together and match the stated target user and the value proposition?

     • unable to judge: You are unable to judge this aspect for some reason.
     • low: One of these
       • target user is too general  i.e. wider than AB4
       • target user and/or value proposition is not clear from the user guide
       • features don't seem to fit together for the most part
     • medium: Some features fit together but some don't.
     • high: All features fit together but the features are not very high value to the target user.
     • excellent: The target user is clearly defined (not too general) and almost all new features are of high-value to the target user. i.e. the product is very attractive to the target user.

   • B. Quality of user docs []: Evaluate based on the parts of the user guide written by the person, as reproduced in the project portfolio. Evaluate from an end-user perspective.

     • unable to judge: Less than 1 page worth of UG content written by the student.
     • low: Hard to understand, often inaccurate or missing important information.
     • medium: Needs some effort to understand; some information is missing.
     • high: Mostly easy to follow. Only a few areas need improvements.
     • excellent: Easy to follow and accurate. Just enough information, visuals, examples etc. (not too much either).

   • C. Quality of developer docs []: Evaluate based on the developer docs cited/reproduced in the respective project portfolio page. Evaluate from the perspective of a new developer trying to understand how the features are implemented.

     • unable to judge: One of these
       • no content at all.
       • less than 0.5 pages worth of content.
       • other problems in the document  e.g. looks like included wrong content.
     • low: One of these
       • Very small amount of content (i.e., 0.5 - 1 page).
       • Hardly any use to the reader (i.e., content doesn't make much sense or redundant).
       • Uses ad-hoc diagrams where UML diagrams could have been used instead.
       • Multiple notation errors in UML diagrams.
     • medium: Some diagrams, some descriptions, but does not help the reader that much  e.g. overly complicated diagrams.
     • high: Enough diagrams (at lest two kinds of UML diagrams used) and enough descriptions (about 2 pages worth) but explanations are not always easy to follow.
     • excellent: Easy to follow. Just enough information (not too much). Minimum repetition of content/diagrams. Good use of diagrams to complement text descriptions. Easy to understand diagrams with just enough details rather than very complicated diagrams that are hard to understand.

   • D. Depth of feature []: Evaluate the feature done by the student for difficulty, depth, and completeness. Note: examples given below assumes AB4 did not have the commands edit, undo, and redo.

     • unable to judge: You are unable to judge this aspect for some reason.
     • low : An easy feature  e.g. make the existing find command case insensitive.
     • medium : Moderately difficult feature, barely acceptable implementation  e.g. an edit command that requires the user to type all fields, even the ones that are not being edited.
     • high: One of the below
       • A moderately difficult feature but fully implemented  e.g. an edit command that allows editing any field.
       • A difficult feature with a reasonable implementation but some aspects are not covered  undo/redo command that only allows a single undo/redo.
     • excellent: A difficult feature, all reasonable aspects are fully implemented  undo/redo command that allows multiple undo/redo.

   • E. Amount of work []: Evaluate the amount of work, on a scale of 0 to 30.

     • Consider this PR (history command) as 5 units of effort which means this PR (undo/redo command) is about 15 points of effort. Given that 30 points matches an effort twice as that needed for the undo/redo feature (which was given as an example of an A grade project), we expect most students to be have efforts lower than 20.
     • Consider the main feature only. Exclude GUI inputs, but consider GUI outputs of the feature. Count all implementation/testing/documentation work as mentioned in that person's portfolio page. Also look at the actual code written by the person. We understand that it is not possible to know exactly which part of the code is for the main feature; make a best guess judgement call based on the available info.
     • Do not give a high value just to be nice. If your estimate is wildly inaccurate, it means you are unable to estimate the effort required to implement a feature in a project that you are supposed to know well at this point. You will lose marks if that is the case.

Bug Review Period:

There will be a review period for you to respond to the bug reports you received.

Duration: The review period will start around 1 day after the PE (exact time to be announced) and will last until the following Tuesday midnight.

Bug reviewing is recommended to be done as a team as some of the decisions need team consensus.

Instructions for Reviewing Bug Reports

• First, don't freak out if there are lot of bug reports. Many can be duplicates and some can be false positives. In any case, we anticipate that all of these products will have some bugs and our penalty for bugs is not harsh. Furthermore, it depends on the severity of the bug. Some bug may not even be penalized.

• Do not edit the subject or the description. Do not close bug reports. Your response (if any) should be added as a comment.

• If the bug is reported multiple times, mark all copies EXCEPT one as duplicates using the duplicate tag (if the duplicates have different severity levels, you should keep the one with the highest severity). In addition, use this technique to indicate which issue they are duplicates of. Duplicates can be omitted from processing steps given below.

• If a bug seems to be for a different product (i.e. wrongly assigned to your team), let us know (email prof).

• Decide if it is a real bug and apply ONLY one of these labels.

Response Labels:

• response.Accepted : You accept it as a bug.
• response.Rejected : What tester thought as a bug is in fact expected behavior. ❗️ The penalty for rejecting a bug using an unjustifiable explanation is higher than the penalty if the same bug was accepted. You can also reject bugs that you inherited from AB4.
• response.CannotReproduce : You are unable to reproduce the behavior reported in the bug after multiple tries.
• response.IssueUnclear : The issue description is not clear.

• If applicable, decide the type of bug. Bugs without type- are considered type-FunctionalityBug by default (which are liable to a heavier penalty):

Bug Type Labels:

• type-FunctionalityBug : the bug is a flaw in how the product works.
• type-DocumentationBug : the bug is in the documentation.

• If you disagree with the original severity assigned to the bug, you may change it to the correct level, in which case add a comment justifying the change. All such changes will be double-checked by the teaching team and unreasonable lowering of severity will be penalized extra.:

Bug Severity labels:

• severity.Low : A flaw that is unlikely to affect normal operations of the product. Appears only in very rare situations and causes a minor inconvenience only.
• severity.Medium : A flaw that causes occasional inconvenience to some users but they can continue to use the product.
• severity.High : A flaw that affects most users and causes major problems for users. i.e., makes the product almost unusable for most users.

• Decide who should fix the bug. Use the Assignees field to assign the issue to that person(s). There is no need to actually fix the bug though. It's simply an indication/acceptance of responsibility. If there is no assignee, we will distribute the penalty for that bug (if any) among all team members.

• Add an explanatory comment explaining your choice of labels and assignees.

At the end of the project each student is required to submit a Project Portfolio Page.

• Objective:

  • For you to use  (e.g. in your resume) as a well-documented data point of your SE experience
  • For us to use as a data point to evaluate your,
    • contributions to the project
    • your documentation skills

• Sections to include:

  • Overview: A short overview of your product to provide some context to the reader.

  • Summary of Contributions:

    • Code contributed: Give a link to your code on Project Code Dashboard, which should be https://nuscs2113-ay1819s1.github.io/dashboard/#=undefined&search=githbub_username_in_lower_case (replace githbub_username_in_lower_case with your actual username in lower case e.g., johndoe). This link is also available in the Project List Page -- linked to the icon under your photo.
    • Main feature implemented: A summary of the main feature you implemented
    • Other contributions:
      • Contributions to project management e.g., setting up project tools, managing releases, managing issue tracker etc.
      • Evidence of helping others e.g. responses you posted in our forum, bugs you reported in other team's products,
      • Evidence of technical leadership e.g. sharing useful information in the forum
    • [Optional] Other minor enhancements: If you have other enhancements that you implemented, which are not related to your main feature, you can include it here. If you have written a significant amount of code that can be advertised as a feature by itself, but does not belong to your main feature, you can choose to include it as a part of the optional enhancements.

  • Contributions to the User Guide: Reproduce the parts in the User Guide that you wrote. This can include features you implemented as well as features you propose to implement.
    The purpose of allowing you to include proposed features is to provide you more flexibility to show your documentation skills. e.g. you can bring in a proposed feature just to give you an opportunity to use a UML diagram type not used by the actual features.

  • Contributions to the Developer Guide: Reproduce the parts in the Developer Guide that you wrote. Ensure there is enough content to evaluate your technical documentation skills and UML modelling skills. You can include descriptions of your design/implementations, possible alternatives, pros and cons of alternatives, etc.

  • If you plan to use the PPP in your Resume, you can also include your SE work outside of the module (will not be graded)

• Format:

  • File name: docs/team/githbub_username_in_lower_case.adoc e.g., docs/team/johndoe.adoc

  • Follow the example in the AddressBook-Level4, but ignore the following two lines in it.
    

    • Minor enhancement: added a history command that allows the user to navigate to previous commands using up/down keys.
    • Code contributed: [Functional code] [Test code] {give links to collated code files}

  • 💡 You can use the Asciidoc's include feature to include sections from the developer guide or the user guide in your PPP. Follow the example in the sample.

  • It is assumed that all contents in the PPP were written primarily by you. If any section is written by someone else  e.g. someone else wrote described the feature in the User Guide but you implemented the feature, clearly state that the section was written by someone else  (e.g. Start of Extract [from: User Guide] written by Jane Doe).  Reason: Your writing skills will be evaluated based on the PPP

  • Page limit: If you have more content than the limit given below, shorten (or omit some content) so that you do not exceed the page limit. Having too much content in the PPP will be viewed unfavorably during grading. Note: the page limits given below are after converting to PDF format. The actual amount of content you require is actually less than what these numbers suggest because the HTML → PDF conversion adds a lot of spacing around content.

    Content	Limit
    Overview + Summary of contributions	0.5-1
    Contributions to the User Guide	1-3
    Contributions to the Developer Guide	3-6
    Total	5-10