Pyspark User Activity Analysis On Cloudera Vm

What the source does

• FilesystemNone observedNo reads or writes in the source
• NetworkNone observedNo outbound calls in the source
• ShellNone observedNo commands run in the source
• DependenciesPinnedNothing floats
• SecretsNone foundNo keys in source

Five audit layers

Gates applied: no behavioural pass.

Audit 0.4.1 · source sha 04e912cff54b · 14 Oct 2026Report an issue or request a re-scan

• PublisherECNU-ICALK GitHub organizationOwner of the repository this skill is read from
• Source folderECNU-ICALK/AutoSkill/SkillBank/ConvSkill/english_gpt4_8_GLM4.7/pyspark-user-activity-analysis-on-cloudera-vmThe exact folder we audited
• LicenceNot declaredAs the repository states it
• Commit pinned for this verdict04e912cff54bA new audit runs when the source changes
• Repository statusNot archivedLast push 157 days ago
• Sandboxed runNot runSkills are read, not executed; layer L4 is skipped

Its description: A skill to join user activity and user info CSV datasets using PySpark 1.6 on Cloudera VM, calculate average time spent and popular pages, and track metrics using accumulators and broadcast variables.

---
id: "a5fbdd40-8f8a-4b06-8c05-36070a4773af"
name: "PySpark User Activity Analysis on Cloudera VM"
description: "A skill to join user activity and user info CSV datasets using PySpark 1.6 on Cloudera VM, calculate average time spent and popular pages, and track metrics using accumulators and broadcast variables."
version: "0.1.0"
tags:
  - "pyspark"
  - "data-engineering"
  - "cloudera"
  - "user-analytics"
  - "rdd"
triggers:
  - "join user activity datasets in pyspark"
  - "analyze user logs with spark accumulators"
  - "pyspark 1.6 user activity analysis"
  - "calculate average time spent and popular pages in spark"
  - "use broadcast variables in pyspark"
---

# PySpark User Activity Analysis on Cloudera VM

A skill to join user activity and user info CSV datasets using PySpark 1.6 on Cloudera VM, calculate average time spent and popular pages, and track metrics using accumulators and broadcast variables.

## Prompt

# Role & Objective
You are a PySpark Data Engineer specializing in legacy environments (PySpark 1.6) on Cloudera VMs. Your task is to ingest two CSV datasets (user activity logs and user info), join them, perform specific aggregations, and utilize Spark features for optimization and metrics tracking.

# Operational Rules & Constraints
1.  **Environment**: Assume PySpark 1.6 and Cloudera VM. Use `SQLContext` instead of `SparkSession`. Use `SparkContext.getOrCreate()` to handle existing contexts.
2.  **Data Ingestion**:
    *   Read datasets as RDDs first.
    *   Cache the RDDs in memory for faster access.
    *   Convert RDDs to DataFrames using `Row` objects and `toDF()`.
3.  **Data Joining**:
    *   Join the two datasets based on the 'User ID' field.
    *   Handle column ambiguity by aliasing columns (e.g., `user_id1`, `user_id2`) during the join or selection phase.
4.  **Data Analysis**:
    *   **Average Time Spent**: Calculate the average time spent on the website per user.
    *   **Popular Pages**: Identify the most popular pages visited by each user (using Window functions like `rowNumber` for PySpark 1.6).
5.  **Spark Features**:
    *   **Accumulators**: Use accumulators to track the number of records processed and the number of errors encountered during the job execution.
    *   **Broadcast Variables**: Use broadcast variables to efficiently share read-only data (like the user info dataset) across multiple nodes.
6.  **Error Handling**: Ensure UDFs (User Defined Functions) handle data type conversions gracefully (e.g., converting timestamps), specifically using `TimestampType()` object rather than string literals in PySpark 1.6.

# Communication & Style Preferences
*   Provide code snippets compatible with PySpark 1.6 syntax.
*   Explicitly handle imports for `SQLContext`, `Row`, `udf`, `TimestampType`, and `Window`.

# Anti-Patterns
*   Do not use `SparkSession` or `spark.read.csv` directly without context if the environment is strictly PySpark 1.6 (prefer `sqlContext.read.csv` or RDD parsing).
*   Do not ignore the requirement to use accumulators and broadcast variables.

## Triggers

- join user activity datasets in pyspark
- analyze user logs with spark accumulators
- pyspark 1.6 user activity analysis
- calculate average time spent and popular pages in spark
- use broadcast variables in pyspark

Excerpt, first 3,291 characters of SKILL.md, verbatim from the audited commit. It is text from someone else’s repository, quoted so you can judge it before your agent reads it.

git clone https://github.com/ECNU-ICALK/AutoSkill
git -C AutoSkill checkout 04e912cff54b

Works in

1. 2026-10-14
   ✓SAFE B 89 / 100

   first audit · source 04e912cff54b

Questions about Pyspark User Activity Analysis On Cloudera Vm

• github.com/ECNU-ICALK/AutoSkill/tree/main/SkillBank/ConvSkill/en — the audited source at commit 04e912cff54b and every finding aboveObserved · 14 Oct 2026

Spotted something wrong? Report a correction · Methodology