symfony advent calendar day twenty: Administration and moderation ================================================================= Previously on symfony --------------------- The askeet service should work as expected and without any bad surprises, thanks to our concern about performance before the initial release. But there is a much bigger problem: being an application open to contributions from anyone, it is subject to spam, excesses, or disturbing errors. Every service like askeet needs a way to moderate the publications, and accessing the database by hand is surely a bad solution. Should we add a backend application to askeet? The advent calendar tutorials are supposed to talk about the development of a web application using agile methods. However, until now, we talked a lot about coding and not that much about application development, and the relations between the requirements of a client and the functionality implemented. The backend need will be a good opportunity to illustrate what comes before coding in agile development. The expected result: what the client says ----------------------------------------- Today's job will consist of a few new actions, new templates and new model method, and we already know how to do that. The hardest part is probably to define what is needed, and where to put it. It is both a functional and usability concern, and it's a good thing that developers focus on something else than code every once in a while. It will be the opportunity to illustrate one of the tasks of the [eXtreme Programming (XP) methodology](http://www.extremeprogramming.org/what.html): the writing of **stories**, and the work that developers have to do to transform stories into functionality. XP is one of the best agile development approaches, and is usually applicable to web 2.0 projects like askeet. ### Stories In XP, a story is a brief description of the way an action of the user triggers a reaction of the application. Stories are written by the client of the website (the one who eventually pays for it - the web is not all about open source). Stories rarely exceed one or two sentences. They are regrouped in themes. The stories are generally less detailed and more elementary than use cases. If you are familiar with UML, you might find the stories to not be precise enough, but we will see shortly that it can be a great chance. Stories focus on the result of the action, not the implementation details. Of course, the client may have preferences concerning the interface, and in this case the story has to contain the demands and recommendations about the look and feel of the human computer interaction. Stories have to be small enough to be evaluated easily by developers in terms of development time. Usually, a team of extreme programmers measure stories in units. The value of a unit is refined throughout the course of a project, and can vary from half a day to a few days. Now, let's have a look at how the client would define the requirements for the askeet backend. ### Story #1: Profile management Every user can ask to become a moderator. In a user's profile page, a link should be made available to ask for this privilege. A person who asked to be moderator must not be able to ask it again until he/she receives an answer. The persons entitled to accept or refuse a moderator candidate are the administrators. They must be able to browse the list of candidates, and have a button to grant or refuse the grade of moderator for each one of them. Administrators need to have a link to the candidate's profile to see if their contributions are all right. Granting moderator rights must be a reversible action: Administrators must be able to browse a list of moderators, and for each, to delete the moderator credential. Administrators can also grant administrator rights to other users. They have access to the list of administrators. ### Story #2: Report of problematic questions or answers Every user must be able to report a problematic question or answer to a moderator. A simple 'report spam' link at the bottom of every question or answer can be a good solution. To avoid spam of reports, the report from a user about a specific question or answer can only be counted once. It would be great if the user had a visual feedback about the fact that his/her report was taken into account. ### Story #3: Handling of problematic questions or answers Moderators have two more lists available: the list of problematic questions, and the list of problematic answers. Each list is ordered according to the number of reports, in decreasing order. So the most reported questions will appear on top of the reported question list. Moderators have the ability to delete a question, to delete an answer, and to reset the number of reports about either one. The deletion of a question causes the deletion of all the answers to this question. ### Story #4: handling of problematic tags Moderators have the ability to delete a tag for a question, whether the tag was given by them or not. Moderators have access to a list of tags, ordered by inverse popularity, so that they can detect the problematic tags - the ones that don't make sense. By linking to the list of questions tagged with this tag, the list gives the ability to suppress the tags. ### Story #4: Handling of problematic users When a moderator deletes a user's contribution, it increments the number of problematic contributions posted by this user. Administrators have a list of problematic users ordered by the number of problematic posts erased. Administrators must be able to delete a user and all his/her contributions. ### Is that all? Yes, that's all that the client needs to define about the functionality required for the askeet site management. It doesn't cover all cases as a functional specification would, it is not as accurate as a complete set of use case, and it leaves a lot of open ends that may lead to unwanted results. But the job of the agile developers, which starts now, is to detect the possible ambiguities and lack of data, and to require the assistance of the client when it turns out that a story must be more precise. In a XP-style development phase, the client is always available to answer the questions of the development team. So the developers meet up in pairs, and each pair chooses a story to work on. They talk a bit about what the story means, the unit test cases that would validate the functionality. They write the unit tests. Then, they write the code to pass these tests. When it's done, they release the code that they added in the whole application, and validate the integration by running all the unit tests written before. As it works, they take a cup of coffee, and split up. Then they form a new pair with someone else and focus on a new story. What if the final result doesn't meet up with the desires of the client? Well, it only represents a few units of work (a few hours or days), so it is easy to forget it and try a new approach. At least, the client now knows what he/she *doesn't* want, and that's a great step towards determinism. But most of the time, as the developers are given the opportunity to talk directly with the client and read between the lines of he stories written, they get to produce the functionality in an even better way than the client would expect. Plus, it's the developer who knows about the AJAX possibilities and the way a web 2.0 can become successful. So giving them (us) the initiative is a good chance to end up with a great application. If you are interested in XP and the benefits of agile development, have a look at the [eXtreme Programming website](http://www.extremeprogramming.org/) or read _Extreme Programming Explained: Embrace Change_ by Kent Beck. Backend vs. enhanced frontend ----------------------------- The feedback of the developer on the client's requirements is often crucial for the quality of the application. Let us see what the developer, who knows how the application is built and how powerful symfony is, could say to the client. The idea to add a backend application to askeet is not that good, and for several reasons. First, a moderator using the backend might need a lot of the features already available in the frontend (including the list of latest questions, the login module, etc.). So there is a risk that the backend application repeats part of the frontend. As we don't like to repeat ourselves, that would imply a lot of cross-application refactoring, and this is much too long for the hour dedicated to it. Second, a new application would probably mean a new design to the site, with a custom layout and stylesheets. This is what takes the most time in application development. Last, to create the backend application in one hour, we would probably have to use the CRUD generator a lot, resulting in many unnecessary actions and long-to-adapt templates. In the near future (it is planned for version 0.6), symfony will provide a full-featured back-office generator. All the functionality commonly needed to manage a website activity will be handled easily, almost without a line of code. This brilliant addition would have changed our mind about the way to build the askeet backend, but considering the current state of the framework, the best solution for the management features is to add them to the frontend application. The base of the askeet frontend is a set of lists, and detail pages for questions and users in which certain actions are available. This is exactly the skeleton needed to build up site management functionality on. Although it would be helpful to show how a project can contain more than one application, the client, impressed by this demonstration, goes for an integration of the site management features in the frontend application. >**Note**: If you are still curious about the way to have more than one application running in a symfony project, have a look at the [My first project](http://www.symfony-project.com/tutorial/my_first_project.html) tutorial, which describes it in detail. The functionality: what the developers understand --------------------------------------------------- After the developers meet up and talk with the client about the stories, they deduce the modifications to be done to the askeet application. The developer transforms *stories* to *tasks*. Tasks are usually smaller than stories, because implementing a story takes more than a day or two, while a task can normally be developed within one or two time units. 1. The model has to be modified to allow efficient requests: * new table `ReportQuestion` to be created, with `question_id`, `user_id` and `created_at` columns * new table `ReportAnswer` to be created, with `question_id`, `user_id` and `created_at` columns * new column `reports` to be added to the `Question` and `Answer` tables * new columns `is_administrator`, `is_moderator` and `deletions` to be added to the `User` table 2. On every page, the sidebar has to provide access to new lists according to the credentials of the user: * All users: popular questions, latest questions, latest answers * Moderators: reported questions, reported answers, unpopular tags * Administrators: administrators, moderators, moderator candidates, problematic users 3. The question detail page (`question/show`) has to provide access to new actions according to the credentials of the user: * Subscriber: report question, report answer * Moderators: delete question and answers, delete answer, reset reports for question, reset reports for answer, delete tag The question detail has to give additional information according to the credentials of the user: * Subscriber: if the question has already been reported by the subscriber * Moderator: the number of reports about the question and answers 4. The user profile page (`user/show`) has to provide access to new actions according to the credentials of the user: * Subscriber on his own page: come forward as a moderator candidate * Administrators: delete the user and all his/her contributions, grant moderator credentials, refuse moderator credentials, delete moderator credentials, grant administrator credentials The user profile page has to give additional information according to the credentials of the user: * All users: credentials of the user, credentials being applied for * Administrators: number of erased posts 5. New lists with restricted access must be created: * Restricted to moderators: * `question/reports`: list of reported questions, in decreasing order of number of reports; For each, link to the question detail. * `answer/reports`: list of reported answers, in decreasing order of number of reports; For each, link to the question detail. * `tag/unpopular`: list of tags, in increasing popularity order; For each, link to the list of questions tagged with this tag * Restricted to administrators: * `user/administrators`: list of administrators, by alphabetical order; For each, link to the user profile * `user/moderators`: list of moderators, by alphabetical order; For each, link to the user profile * `user/candidates`: list of moderator candidates, by alphabetical order; For each, link to the user profile * `user/problematic`: list of problematic users, in decreasing order of deleted contributions; For each, link to the user profile 6. Two new credentials must be created: Administrator and Moderator. 7. At least one administrator has to be setup by hand in the database for the application to work. Implementation -------------- Once the task list is written, the way to implement the backend features on askeet with symfony is just a matter of work. Applying the XP methodology on this task list, including the writing of unit tests, would take at least a good day of work. For the needs of the advent calendar tutorial, we will do it a little faster, and we will just focus here on the new techniques not described previously, or on the ones that should help you to review classical symfony techniques. ### New tables For the question and answer reports, we add two new tables to the askeet database: [xml]
The combination of the `question_id`/`answer_id` and the `user id` is enough to create a unique primary key, so we don't need to add an auto-increment `id` for these tables. We also add a new `reports` column to the `Question` and `Answer` table. In order to synchronize the number of records in the `ReportQuestion` and the number of `reports` in the `Question` table, we override the `save()` method of the `ReportQuestion` object to add a transaction, as we did during [day 4](4.txt): [php] public function save($con = null) { $con = sfContext::getInstance()->getDatabaseConnection('propel'); try { $con->begin(); $ret = parent::save(); // update spam_count in answer table $answer = $this->getAnswer(); $answer->setReports($answer->getReports() + 1); $answer->save(); $con->commit(); return $ret; } catch (Exception $e) { $con->rollback(); throw $e; } } Same for the `ReportAnswer` table. ### Cascade deletion When a question is deleted, all the answers to this questions must also be deleted, as well as all the interests about the question, the tags added to the question and the relevancy ratings about all the answers. We need a mechanism of cascade deletion to take care of all that for us. During [day two](2.txt), we had the idea of using the InnoDB engine for the askeet database. This facilitates the cascade deletions. But the Propel layer can manage to do the cascade deletions even on a non-InnoDB enabled database, provided that we indicate in the schema that cascade deletion has to be taken care of. This has to be done when declaring a foreign key: add a `onDelete="cascade"` attribute to the `` tag in a table definition. For instance, for the `Answer` table: [xml] ...
... Once the model is rebuilt, cascade deletion is enabled for the relations bearing the `onDelete` attribute. When you delete a record in the `Question` table: * if the database uses the InnoDB engine, the related answers will be deleted automatically by the database itself * else, the Propel layer will automatically get the related answers, delete them, then delete the question. All relations may not involve a cascade deletion. Deleting a user, for instance, should delete his/her interests and ratings for answer relevancies, but not his/her contributions (questions and answers). These contributions should be associated to the anonymous user after deletion. So the `onDelete` attribute has to be set to `cascade` for the following relations: * `Answer/QuestionId` * `Interest/QuestionId` * `Relevancy/QuestionId` * `QuestionTag/QuestionId` * `ReportQuestion/QuestionId` * `ReportAnswer/AnswerId` ### Add links in the sidebar for users with credentials We create a new `moderator` module to handle all the moderator actions, and an `administrator` one to handle the administration actions. During [day seven](7.txt), we used the component slot technique to store the code of the sidebar in the `sidebar` module. The links to the new lists will appear there, but they need to be conditionned to a credential. This is simply done by using the `$sf_user->hasCredential()` method, as seen during [day six](6.txt): [php] // in askeet/apps/frontend/modules/sidebar/templates/_default.php and _question.php: ... // in askeet/apps/frontend/modules/sidebar/templates/_moderation.php: hasCredential('moderator')): ?>

moderation

// in askeet/apps/frontend/modules/sidebar/templates/_administration.php: ... hasCredential('administrator')): ?>

administration

![new links](/images/askeet/moderation_links.gif) The class methods `QuestionPeer::getReportCount()`, `AnswerPeer::getReportCount()`, `UserPeer::getModeratorCandidatesCount()` and `UserPeer::getProblematicUsersCount()` are to be added to the model. They are all based on the same principle: [php] public static function getReportCount() { $c = new Criteria(); $c->add(self::REPORTS, 0, Criteria::GREATER_THAN); $c = self::addPermanentTagToCriteria($c); return self::doCount($c); } ### AJAX report We will provide a '[report to moderator]' link to report a question in all the places a question is displayed (in the question lists, in a question detail page). It would be nice if this link was an AJAX one, as in the [day eight](8.txt) tutorial. So we add a new helper to the `QuestionHelper.php` file in the `askeet/apps/frontend/lib/helper/` directory: [php] function link_to_report_question($question, $user) { use_helper('Javascript'); $text = '[report to moderator]'; if ($user->isAuthenticated()) { $has_already_reported_question = ReportQuestionPeer::retrieveByPk($question->getId(), $user->getSubscriberId()); if ($has_already_reported_question) { // already reported for this user return '[reported]'; } else { return link_to_remote($text, array( 'url' => '@user_report_question?id='.$question->getId(), 'update' => array('success' => 'report_question_'.$question->getId()), 'loading' => "Element.show('indicator')", 'complete' => "Element.hide('indicator');".visual_effect('highlight', 'report_question_'.$question->getId()), )); } } else { return link_to_login($text); } } Now, the templates where the link has to appear (`question/templates/showSuccess.php`, `question/templates/_list.php`) can use this helper: [php]
The `@user_report_question` rule has to be written in the `routing.yml` as leading to a `user/reportQuestion` action: [php] public function executeReportQuestion() { $this->question = QuestionPeer::retrieveByPk($this->getRequestParameter('id')); $this->forward404Unless($this->question); $spam = new ReportQuestion(); $spam->setQuestionId($this->question->getId()); $spam->setUserId($this->getUser()->getSubscriberId()); $spam->save(); } And the result of this action, the `user/templates/reportQuestionSuccess.php` template, is simply: [php] ![report question](/images/askeet/report_question.gif) The same goes for the reported answers. ### New action links for users with credentials In the `question_body` div tag of the `askeet/apps/frontend/modules/question/templates/showSuccess.php`, we add the question management actions for moderators only, so to be compatible with the AJAX report, we put them in a fragment: [php] ...
$question)) ?>
The `askeet/apps/frontend/modules/moderator/templates/_question_options.php` fragment contains: hasCredential('moderator')): ?> getReports()): ?>  [getReports() ?> reports]  getStrippedTitle()) ?>  getStrippedTitle()) ?> ... ![moderator actions](/images/askeet/question_moderator_options.gif) The same options are added in the `askeet/apps/frontend/modules/answer/templates/_answer.php`, with a link to a `moderator/templates/_answer_options.php` fragment. The same kind of adaptation goes for the administrator action links in the user profile page. >**Note**: One of the good practices about links to actions is to implement them as a normal link (doing a 'GET' request) when the action doesn't modify the model, and as a button (doing a 'POST') request when the action alters the data. This is to avoid that automatic web crawlers, like search engine robots, click on a link that can modify the database. The AJAX links being inmplemented in javascript, they can'y be clicked by robots. The 'reset' and 'report' links that we just added, however, could be clicked by a robot. Fortunately, they are not displayed unless the user has moderator access, so there is no risk that they are clicked unintentionnally. > >We could add an extra protection on these links by declaring them as 'POST' links, as described in the [link chapter](http://www.symfony-project.com/book/1_0/09-Links-and-the-Routing-System) of the symfony book: > > [php] > getId(), 'post=true') ?> > ### Access restriction When a user with specific rights logs in, his `sfUser` object must be given the appropriate credential. This is done in the `signIn` method of the `myUser` class in `askeet/apps/frontend/lib/myUser.class.php`, that we created during [day six](6.txt): [php] public function signIn($user) { $this->setAttribute('subscriber_id', $user->getId(), 'subscriber'); $this->setAuthenticated(true); $this->addCredential('subscriber'); if ($user->getIsModerator()) { $this->addCredential('moderator'); } if ($user->getIsAdministrator()) { $this->addCredential('administrator'); } $this->setAttribute('nickname', $user->getNickname(), 'subscriber'); } Of course, all the moderator actions have to be restricted to moderators with appropriate settings in the `askeet/apps/frontend/modules/moderator/config/security.yml`: all: is_secure: on credentials: moderator The same kind of restriction is to be applied for administrator actions. ### New `moderator` and `administrator` actions There is nothing new in the actions to be added to the `moderator` and `administrator` actions. We will just give the list here so that you know about them: // administrator actions executeProblematicUsers() -> usersSuccess.php executeModerators() -> usersSuccess.php executeAdministrators() -> usersSuccess.php executeModeratorCandidates() -> usersSuccess.php executePromoteModerator() -> request referrer executeRemoveModerator() -> request referrer executePromoteAdministrator() -> request referrer executeRemoveAdministrator() -> request referrer // moderator actions executeUnpopularTags() -> unpopularTagsSuccess.php executeReportedQuestions() -> reportedQuestions.php executeReportedAnswers() -> reportedAnswers.php executeDeleteTag() -> request referrer executeDeleteQuestion() -> @homepage executeDeleteAnswer() -> request referrer >**Note**: To specify a custom template for an action, you can add a `view.yml` config file to the module. For instance, to have half of the `administrator` actions use the `usersSuccess.php` template, you can create the following `askeet/apps/frontend/modules/administrator/config/view.yml` file: > > moderatorsSuccess: > template: users > > administratorsSuccess: > template: users > > moderatorCandidatesSuccess: > template: users > > problematicUsersSuccess: > template: users > ### Log deletions When a moderator deletes a question, we want to keep a trace of the deletion in a log file, in a warning message. To allow the logging of warning messages in the production environment, we need to modify the `logging.yml` configuration file: prod: level: warning Then, in all the delete actions, add the code to log the deletion, as in this `moderator/deleteQuestion` action: [php] public function executeDeleteQuestion() { $question = QuestionPeer::getQuestionFromTitle($this->getRequestParameter('stripped_title')); $this->forward404Unless($question); $con = sfContext::getInstance()->getDatabaseConnection('propel'); try { $con->begin(); $user = $question->getUser(); $user->setDeletions($user->getDeletions() + 1); $user->save(); $question->delete(); $con->commit(); // log the deletion $log = 'moderator "%s" deleted question "%s"'; $log = sprintf($log, $this->getUser()->getNickname(), $question->getTitle()); $this->getContext()->getLogger()->warning($log); } catch (PropelException $e) { $con->rollback(); throw $e; } $this->redirect('@homepage'); } If you want to know more about logging, you can have a look at the [debug chapter](http://www.symfony-project.com/book/1_0/16-Application-Management-Tools) of the symfony book. We changed the `try/catch` statement to react only to `PropelExceptions` instead of all `Exceptions`. This is because we don't want the transaction to fail only because there is a problem in the logging of the deletion. >**Note**: In the example above, we use the object `$question` *even after it has been deleted*. This is because the call to the `->delete()` method marks a record or a list of records for deletion, and the actual deletion is only processed by Propel once the action is finished. See you Tomorrow ---------------- As we took some time to think about the way to implement the backend features, and because there are quite a lot of them, today's tutorial probably lasted two hours rather than only one. But there is not many new things here, so the implementation should be a review of symfony techniques. You can have a good view of the total list of changes by browsing to the [askeet timeline](http://trac.askeet.com/trac/changeset/55). Tomorrow is the day of the mysterious feature. Numerous suggestions were sent to the forum, or even in the beta askeet site itself. You will see which one we decided to implement, and how symfony can be of great help to it. Feel free to go to the [forum](http://www.symfony-project.com/forum/index.php/f/8/) if you have any problem with today's source, which, by the way, can still be downloaded from the [SVN repository](http://svn.askeet.com/tags/release_day_14/) or browsed in the [trac](http://trac.askeet.com/trac/browser/tags/release_day_20).