{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Evaluación del clasificador desde ciberseguridad\n",
        "## Objetivo del laboratorio\n",
        "Evaluar un clasificador de amenazas textuales mediante métricas como precision, recall, F1-score y matriz de confusión, analizando además el efecto de los umbrales de alerta y el costo del error en contextos reales de ciberseguridad.\n"
      ],
      "metadata": {
        "id": "83VgXI2SLO2z"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Parte 1 — Preparación del entorno: Importar librerías\n",
        "\n",
        "Las librerías pandas y numpy permitirán:\n",
        "- vectorizar textos,\n",
        "- entrenar un clasificador,\n",
        "- calcular métricas,\n",
        "- analizar errores."
      ],
      "metadata": {
        "id": "B4XICXduLuJs"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "WqU-JWI8Ivqc"
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "\n",
        "\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "from sklearn.linear_model import LogisticRegression\n",
        "\n",
        "from sklearn.metrics import (\n",
        "    classification_report,\n",
        "    confusion_matrix,\n",
        "    precision_score,\n",
        "    recall_score,\n",
        "    f1_score\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Parte 2 — Crear un dataset de ejemplo (conjunto de datos simple)\n",
        "Este dataset es pequeño, pero suficiente para practicar interpretación de métricas."
      ],
      "metadata": {
        "id": "euZOiLkqMEAZ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "data = [\n",
        "    (\"URGENTE verifique su cuenta ahora [URL]\", \"phishing\"),\n",
        "    (\"actualice sus credenciales inmediatamente [URL]\", \"phishing\"),\n",
        "    (\"confirmacion requerida para evitar suspension [URL]\", \"phishing\"),\n",
        "    (\"hemos detectado actividad sospechosa confirme su acceso\", \"phishing\"),\n",
        "\n",
        "    (\"no puedo acceder al VPN desde ayer\", \"legitimo\"),\n",
        "    (\"error de autenticacion en el sistema ERP\", \"legitimo\"),\n",
        "    (\"ticket abierto por falla del servidor\", \"legitimo\"),\n",
        "    (\"problema con conexion interna y acceso remoto\", \"legitimo\"),\n",
        "\n",
        "    (\"recordatorio reunion semanal del equipo\", \"legitimo\"),\n",
        "    (\"newsletter corporativo mensual\", \"legitimo\"),\n",
        "]\n",
        "\n",
        "df = pd.DataFrame(data, columns=[\"texto\", \"etiqueta\"])\n",
        "print(df)"
      ],
      "metadata": {
        "id": "Je5OWNznMGVF"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Parte 3 — Vectorización y división de datos:\n",
        "- Convertir texto a TF-IDF\n",
        "- Dividir entrenamiento y prueba"
      ],
      "metadata": {
        "id": "2rmCl4KdMTPb"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Texto a TF-IDF\n",
        "vectorizer = TfidfVectorizer()\n",
        "X = vectorizer.fit_transform(df[\"texto\"])\n",
        "y = df[\"etiqueta\"]\n",
        "\n",
        "#Dividir entrenamiento y prueba\n",
        "X_train, X_test, y_train, y_test = train_test_split(\n",
        "    X, y,\n",
        "    test_size=0.3,\n",
        "    stratify=y,\n",
        "    random_state=42\n",
        ")"
      ],
      "metadata": {
        "id": "8QEJhCEkMkDv"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Parte 4 — Entrenar el clasificador: Crear y entrenar el modelo\n",
        "Usaremos Logistic Regression porque permite obtener probabilidades, necesarias para trabajar con umbrales."
      ],
      "metadata": {
        "id": "92lMyYTWMyFY"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model = LogisticRegression()\n",
        "model.fit(X_train, y_train)"
      ],
      "metadata": {
        "id": "Eu4Np9OTM6Xo"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Parte 5 — Predicción y métricas básicas"
      ],
      "metadata": {
        "id": "vRZri7-pNCgt"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Predecir clases\n",
        "y_pred = model.predict(X_test)\n",
        "\n",
        "#Calcular precision, recall y F1-score\n",
        "precision = precision_score(y_test, y_pred, pos_label=\"phishing\")\n",
        "recall = recall_score(y_test, y_pred, pos_label=\"phishing\")\n",
        "f1 = f1_score(y_test, y_pred, pos_label=\"phishing\")\n",
        "\n",
        "print(\"Precision:\", precision)\n",
        "print(\"Recall:\", recall)\n",
        "print(\"F1-score:\", f1)\n",
        "\n",
        "#Generar reporte completo\n",
        "print(classification_report(y_test, y_pred))\n",
        "\n",
        "#Matriz de confusión\n",
        "cm = confusion_matrix(y_test, y_pred, labels=[\"legitimo\", \"phishing\"])\n",
        "print(cm)"
      ],
      "metadata": {
        "id": "lgZ3R5T8NEEu"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Parte 6 — Análisis del impacto de errores en SOC\n",
        "\n",
        "Identificar falsos positivos y negativos"
      ],
      "metadata": {
        "id": "BKQXZPw3PqS6"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "X_text_train, X_text_test, y_train, y_test = train_test_split(\n",
        "    df[\"texto\"], y,\n",
        "    test_size=0.3,\n",
        "    stratify=y,\n",
        "    random_state=42\n",
        ")\n",
        "\n",
        "X_train = vectorizer.fit_transform(X_text_train)\n",
        "X_test = vectorizer.transform(X_text_test)\n",
        "\n",
        "model = LogisticRegression()\n",
        "model.fit(X_train, y_train)\n",
        "y_pred = model.predict(X_test)\n",
        "\n",
        "resultados = pd.DataFrame({\n",
        "    \"texto\": X_text_test.values,\n",
        "    \"real\": y_test.values,\n",
        "    \"pred\": y_pred\n",
        "})\n",
        "\n",
        "print(resultados)"
      ],
      "metadata": {
        "id": "FrLFqT6nP_Ac"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Parte 7 — Umbrales de alerta"
      ],
      "metadata": {
        "id": "PwyZ0XQjQJ9A"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Obtener probabilidades\n",
        "probs = model.predict_proba(X_test)\n",
        "clases = model.classes_\n",
        "print(clases)\n",
        "\n",
        "#Buscar cuál columna corresponde a \"phishing\":\n",
        "phishing_idx = list(clases).index(\"phishing\")\n",
        "phishing_probs = probs[:, phishing_idx]\n",
        "print(phishing_probs)\n",
        "\n",
        "#Aplicar un umbral personalizado\n",
        "threshold = 0.4\n",
        "#Probar con cada uno de los siguientes:\n",
        "#threshold = 0.3\n",
        "#threshold = 0.5\n",
        "#threshold = 0.7\n",
        "\n",
        "\n",
        "y_pred_threshold = np.where(phishing_probs >= threshold, \"phishing\", \"legitimo\")\n",
        "print(y_pred_threshold)\n",
        "\n",
        "\n",
        "#Comparar métricas con el nuevo umbral\n",
        "precision_t = precision_score(y_test, y_pred_threshold, pos_label=\"phishing\")\n",
        "recall_t = recall_score(y_test, y_pred_threshold, pos_label=\"phishing\")\n",
        "f1_t = f1_score(y_test, y_pred_threshold, pos_label=\"phishing\")\n",
        "\n",
        "print(\"Umbral:\", threshold)\n",
        "print(\"Precision:\", precision_t)\n",
        "print(\"Recall:\", recall_t)\n",
        "print(\"F1-score:\", f1_t)"
      ],
      "metadata": {
        "id": "FlsbOP10QNKs"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Parte 8 — Costo del error: Simular costo operacional\n",
        "Supongamos:\n",
        "- Falso positivo = costo 1\n",
        "- Falso negativo = costo 10\n",
        "\n",
        "Comparar el costo total para varios umbrales y decidir:\n",
        "¿qué umbral reduce mejor el riesgo?\n"
      ],
      "metadata": {
        "id": "K7rG_33XRF__"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "cm = confusion_matrix(y_test, y_pred_threshold, labels=[\"legitimo\", \"phishing\"])\n",
        "\n",
        "TN, FP, FN, TP = cm.ravel()\n",
        "\n",
        "costo_total = FP * 1 + FN * 10\n",
        "\n",
        "print(\"FP:\", FP)\n",
        "print(\"FN:\", FN)\n",
        "print(\"Costo total:\", costo_total)"
      ],
      "metadata": {
        "id": "6FMWqzINRLFz"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}